Nicolas,

I agree that the situation with 1,000 or 100,000 of scaffolds is extremely
important for non-model organism work.  I have had the situation with Honey
bee with roughly 10,000 scaffolds.  And this indeed used to slow down
GenomicRanges to a crawl.  It was essentially impossible to get work done
and I had to just exclude all small scaffolds.  However, this should be
better with the recent addition of IntervalTrees.  I have done some very
simple benchmarking which suggests we are doing much, much better in terms
of speed, but I would not be surprised if there are still opportunities for
improvement.

This situation is something that I think should very much be in the back of
the mind of anyone doing code development.

Best,
Kasper


On Thu, Dec 5, 2013 at 7:40 AM, Nicolas Delhomme <nicolas.delhomme@umu.se>wrote:

> Hej all!
>
> Just my 2 cents. There will eventually be chromosome larger than 2GB.
> Spruce has 12 chromosomes evenly sized for a genome of ~20GB and that’s not
> the largest of the Gymnosperms genomes. Pinus taeda  - currently being
> sequenced - has a genome around 24GB, again with evenly sized chromosomes,
> so that’s getting closed to 2GB :-). But because these genomes are
> excessively repetitive, we are nowhere close to have a full chromosome
> sequence and this I believe for some more time - that time being years
> would not come as a surprise to me.  So I think there’s still time before
> this becomes an issue, and by then the IT would have evolved further.
>
> On the other hand, what might turn into a problem earlier is that these
> released genomes are draft made of literally millions of scaffolds; hence
> very different from what most model organisms are, being constituted of a
> few handfuls of chromosomes. Hervé, Martin do you think this could be an
> issue for IRanges / GenomicRanges / Rsamtools? I’ve now got a number of BAM
> files aligned to Norway Spruce - half a million scaffolds - that I can
> share if you wish to use them for benchmarks.
>
> Next step in my pipeline is anyway to analyse these alignments, so I’m
> bound to see how it works out in R :-)
>
> Cheers,
>
> Nico
>
> ---------------------------------------------------------------
> Nicolas Delhomme
>
> Genome Biology Computational Support
>
> European Molecular Biology Laboratory
>
> Tel: +49 6221 387 8310
> Email: nicolas.delhomme@embl.de
> Meyerhofstrasse 1 - Postfach 10.2209
> 69102 Heidelberg, Germany
> ---------------------------------------------------------------
>
>
>
>
>
> On 03 Dec 2013, at 21:41, Hervé Pagès <hpages@fhcrc.org> wrote:
>
> > Hi Kasper,
> >
> > On 12/03/2013 12:25 PM, Kasper Daniel Hansen wrote:
> >> Is integer.max dependent on 32bit vs 64bit?
> >
> > I don't think so. AFAIK integers are always 32-bit in R (at least on
> > Intel platforms), even on 64-bit OSes. So .Machine$integer.max is
> > always 2^31 - 1 (roughly 2 billions).
> >
> >> It seems to me that the OP
> >> specifically complains that he cannot represent 995*10^6 as an integer.
> >
> > 995*10^6 is roughly 1 billion so it can be represented as an integer,
> > except maybe on some exotic systems.
> >
> >> Also, is there a sign issue here as well?
> >
> > Not that I know of.
> >
> > H.
> >
> >>
> >>
> >> On Tue, Dec 3, 2013 at 2:53 PM, Hervé Pagès <hpages@fhcrc.org
> >> <mailto:hpages@fhcrc.org>> wrote:
> >>
> >>   Hi,
> >>
> >>   Agreed with Martin that until someone comes up with a chromosome that
> >>   is longer than .Machine$integer.max I don't see the need for switching
> >>   to double or int64 to represent the seqlengths.
> >>
> >>   Furthermore, since the seqlengths are used in many range operations
> >>   like checking the validity of the ranges in a GRanges object, trimming
> >>   them, computing coverage, handling circularity, etc... it would not
> >>   make much sense to make the switch for the seqlengths without also
> >>   making it for Ranges objects. That would be a serious undertaking
> though
> >>   and probably with many backward compatibility issues.
> >>
> >>   H.
> >>
> >>
> >>
> >>   On 12/03/2013 10:07 AM, Martin Morgan wrote:
> >>
> >>       On 12/03/2013 02:29 AM, Julian Gehring wrote:
> >>
> >>           Hi,
> >>
> >>           Some of the chromosomes out in the world are fairly large
> >>           (e.g. wheat
> >>           chr 3B
> >>           with > 995 Mbp [1]).  Currently, the 'seqlengths' of the
> >>           reference
> >>           sequence are
> >>           stored as 'integers' which do not allow to store lengths of
> this
> >>           size.  Are
> >>           there any plans of switching to 'doubles' or 64-bit integers
> >>           for the
> >>           'seqlengths' slot?  Or extending the slot such that a user
> >>           can store
> >>           it either
> >>           as integer or floating-point number?
> >>
> >>
> >>       But
> >>
> >>> .Machine$integer.max
> >>       [1] 2147483647 <tel:%5B1%5D%202147483647>
> >>
> >>       so we at least survive wheat chr 3B?
> >>
> >>       If there is movement to support this I'd encourage exact
> >>       representation
> >>       as double (this is how R deals with long vectors, and I believe
> >>       it is
> >>       the javascript representation of integers so not completely
> >>       unprecedented) rather than 64 bit integers (which do not have any
> >>       support in R).
> >>
> >>       I guess this would be quite a big undertaking so real use cases
> >>       need to
> >>       be present. And support for larger integers would seem to be
> >>       useful to R
> >>       generally rather than just to Bioc.
> >>
> >>       Martin
> >>
> >>
> >>           Best wishes
> >>           Julian
> >>
> >>
> >>           [1] http://www.sciencemag.org/__content/322/5898/101
> >>           <http://www.sciencemag.org/content/322/5898/101>
> >>
> >>           _________________________________________________
> >>           Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org>
> >>           mailing list
> >>           https://stat.ethz.ch/mailman/__listinfo/bioc-devel
> >>           <https://stat.ethz.ch/mailman/listinfo/bioc-devel>
> >>
> >>
> >>
> >>
> >>   --
> >>   Hervé Pagès
> >>
> >>   Program in Computational Biology
> >>   Division of Public Health Sciences
> >>
> >>   Fred Hutchinson Cancer Research Center
> >>   1100 Fairview Ave. N, M1-B514
> >>   P.O. Box 19024
> >>   Seattle, WA 98109-1024
> >>
> >>   E-mail: hpages@fhcrc.org <mailto:hpages@fhcrc.org>
> >>   Phone: (206) 667-5791 <tel:%28206%29%20667-5791>
> >>   Fax: (206) 667-1319 <tel:%28206%29%20667-1319>
> >>
> >>
> >>   _________________________________________________
> >>   Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org> mailing
> list
> >>   https://stat.ethz.ch/mailman/__listinfo/bioc-devel
> >>   <https://stat.ethz.ch/mailman/listinfo/bioc-devel>
> >>
> >>
> >
> > --
> > Hervé Pagès
> >
> > Program in Computational Biology
> > Division of Public Health Sciences
> > Fred Hutchinson Cancer Research Center
> > 1100 Fairview Ave. N, M1-B514
> > P.O. Box 19024
> > Seattle, WA 98109-1024
> >
> > E-mail: hpages@fhcrc.org
> > Phone:  (206) 667-5791
> > Fax:    (206) 667-1319
> >
> > _______________________________________________
> > Bioc-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
> ---------------------------------------------------------------
> Nicolas Delhomme
>
> Nathaniel Street Lab
> Department of Plant Physiology
> Umeå Plant Science Center
>
> Tel: +46 90 786 7989
> Email: nicolas.delhomme@plantphys.umu.se
> SLU - Umeå universitet
> Umeå S-901 87 Sweden
> ---------------------------------------------------------------
>
>

	[[alternative HTML version deleted]]

