Nicolas,

Experiment data packages serve different needs, including (1) a small
dataset for testing / running examples for a specific package.  (2) a
large(r) dataset for teaching purposes and/or to showcase a real analysis
workflow.  You seem to suggest that your data package is tightly integrated
with your "analysis" package.  In general, as the package gets bigger, I
think you do have a responsibility to make it useful to other analysis
packages, so that several analysis packages can plug into it.

In this specific case, I would make sure the IDATs are deposited in a
public database, and then I would only include rda's in the data package,
together with a scrip which retrieves and packages the IDATs into the
object in the data package.

Best,
Kasper


On Fri, Nov 8, 2013 at 2:00 PM, Sean Davis <sdavis2@mail.nih.gov> wrote:

> On Fri, Nov 8, 2013 at 1:54 PM, Nicolas De Jay <
> nicolas.dejay@mail.mcgill.ca
> > wrote:
>
> > In that case, I will try to see if the public databases have the kind
> > of data sets I am trying to package and run the idea by the team that
> > is assigned to the project I am developing.  Thank you Martin, Sean
> > and Kasper for your valuable insight!
> >
> >
> [EDITORIAL COMMENT]
> No matter what you decide about the experiment data package, if you have
> data that you are willing to share (and it sounds like you do), putting
> them in a public repository is a GOOD THING.
> [END EDITORIAL COMMENT]
>
> Sean
>
>
>
> > ---
> > Nicolas De Jay
> >
> > On Fri, Nov 8, 2013 at 9:07 AM, Sean Davis <sdavis2@mail.nih.gov> wrote:
> > >
> > >
> > >
> > > On Fri, Nov 8, 2013 at 8:41 AM, Martin Morgan <mtmorgan@fhcrc.org>
> > wrote:
> > >>
> > >> On 11/07/2013 09:26 PM, Nicolas De Jay wrote:
> > >>>
> > >>> Thanks for the prompt answer.  The data set I am packaging closely
> > >>> resembles that of minfiData except that there are 52 samples; the
> IDAT
> > >>> files together are some 800MB whereas the Rda file is closer to
> 150MB.
> > >>>   It is worth noting that my experiment data package will be
> submitted
> > >>> to Bioconductor along with a software package which makes use of
> these
> > >>> samples in the vignette.  With this in mind, can I omit the IDAT
> > >>> files?  If this goes against Bioconductor's underlying design, what
> > >>> would you say is the maximum size of an experiment data package?
> > >>
> > >>
> > >> Hi Nicolas -- Some things to bear in mind.
> > >>
> > >
> > > Hi, Nicolas.
> > >
> > > I just wanted to note that experiment data packages are meant as a
> > > convenient way to distribute data so that reproducible workflows and
> > > documentation can be created easily.  There are other options such as
> > > accessing the data directly from public repositories using Bioconductor
> > > tools that serve the same purpose.  While accessing such online
> resources
> > > does necessitate a one-time network connection (after which packages
> like
> > > GEOquery can use locally cached data), when appropriate datasets exist
> in
> > > public repositories, it may be a perfectly viable alternative to
> > experiment
> > > data packages.  In this particular case, as of today in NCBI GEO, there
> > are
> > > 1711 Human 450k samples with IDAT files available.  I am not arguing
> that
> > > this route should replace experiment data packages, just that stable
> > public
> > > data resources are an alternative to them to consider.
> > >
> > > Sean
> > >
> > >
> > >>
> > >> Files are compressed in package tar balls, so your IDAT files may
> have a
> > >> considerably smaller effective size.
> > >>
> > >> Generally, original text files are a much better way to store external
> > >> data than Rda files. For instance, rda files require updating when /
> if
> > the
> > >> class definition changes, and the provenance and content of the data
> is
> > >> unambiguous.
> > >>
> > >> Experiment data packages are meant to provide reusable examples for
> > >> pedagogic purposes. One would hope that minfiData fulfills this
> > requirement.
> > >> If not, then it would be better to continue the current discussion
> with
> > >> Kasper and others in the community to identify an appropriately
> > >> comprehensive data set for use across many relevant packages.
> > >>
> > >> There is no formal statement about the maximum size of experiment data
> > >> packages, but one would need to make a strong argument for why a Gb of
> > >> experiment data is necessary (including why existing experiment data
> > >> packages are fundamentally inadequate), especially if it is to
> support a
> > >> single package.
> > >>
> > >> Martin
> > >>
> > >>>
> > >>> ---
> > >>> Nicolas De Jay
> > >>>
> > >>> On Thu, Nov 7, 2013 at 9:38 PM, Kasper Daniel Hansen
> > >>> <kasperdanielhansen@gmail.com> wrote:
> > >>>>
> > >>>> To give some background: it is true that the RGsetEx object (in
> > >>>> data/RGsetEx.rda) is a 1-1 correspondence with the raw data files in
> > >>>> inst/extdata, so one could consider it redundant.  However, having
> the
> > >>>> IDAT
> > >>>> files are convenient for testing parsing, and also for other tools
> who
> > >>>> want
> > >>>> to have 450k example data and not want to depend on minfi.  Those
> are
> > >>>> the
> > >>>> two main reasons for including the raw data as well.  And then the
> > fact
> > >>>> that
> > >>>> while the data size is "big" it is only 6 samples.
> > >>>>
> > >>>> Best,
> > >>>> Kasper
> > >>>>
> > >>>>
> > >>>> On Thu, Nov 7, 2013 at 3:58 PM, Nicolas De Jay
> > >>>> <nicolas.dejay@mail.mcgill.ca> wrote:
> > >>>>>
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> I am preparing a data package and using the minfiData package as a
> > >>>>> reference.  The .idat files in extdata and the .rda file in data
> are
> > >>>>> both present in both the compressed tarball source and the
> installed
> > >>>>> copy directory (in my case, under ~/R/x86-64.../3.0/minfiData).
> >  Isn't
> > >>>>> this redundant?  Is there a way to have the prospective user only
> > >>>>> download the .rda files?
> > >>>>>
> > >>>>> Sorry if my question is misguided and thanks in advance for your
> > help.
> > >>>>>
> > >>>>> ---
> > >>>>> Nicolas De Jay
> > >>>>> M.Sc. Student
> > >>>>> Department of Human Genetics
> > >>>>> Montreal Children's Hospital Research Institute, McGill University
> > >>>>> Health
> > >>>>> Centre
> > >>>>> 4060 Ste Catherine West, PT-239
> > >>>>> Montreal, QC H3Z2Z3, Canada
> > >>>>> T: (514) 412-4440 | E: nicolas.dejay@mail.mcgill.ca
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> Bioc-devel@r-project.org mailing list
> > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>> _______________________________________________
> > >>> Bioc-devel@r-project.org mailing list
> > >>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > >>>
> > >>
> > >>
> > >> --
> > >> Computational Biology / Fred Hutchinson Cancer Research Center
> > >> 1100 Fairview Ave. N.
> > >> PO Box 19024 Seattle, WA 98109
> > >>
> > >> Location: Arnold Building M1 B861
> > >> Phone: (206) 667-2793
> > >>
> > >> _______________________________________________
> > >> Bioc-devel@r-project.org mailing list
> > >> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > >
> > >
> >
> > _______________________________________________
> > Bioc-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>

	[[alternative HTML version deleted]]

