On Wed, Apr 16, 2014 at 3:15 PM, Hervé Pagès <hpages@fhcrc.org> wrote:

> Hi Michael,
>
>
> On 04/16/2014 02:33 PM, Michael Lawrence wrote:
>
>> Another interesting aspect of 2bit is that it supports simple masking.
>> I'm all for the 2bit direction. But then BSgenome would depend on
>> rtracklayer?
>>
>
> Any reason I should not do that? ;-)
>
>
>
Because rtracklayer depends on BSgenome ;) At least one reason is to get
the Seqinfo for a named genome. Maybe GenomeInfoDb or something could
provide that instead? It does have a fallback to query UCSC directly, but
that depends on network access. There is also an export method for BSgenome
to TwoBitFile, which would obviously move.



>  Or have you reimplemented it?
>>
>
> Nope.
>
> H.
>
>
>>
>> On Wed, Apr 16, 2014 at 12:59 PM, Hervé Pagès <hpages@fhcrc.org
>> <mailto:hpages@fhcrc.org>> wrote:
>>
>>     Hi Sean
>>
>>     [hope you don't mind if I cc Bioc-devel]
>>
>>     On 04/15/2014 11:47 PM, Maintainer wrote:
>>
>>         Hi The Bioconductor Dev Team,
>>         A new package called BSgenome.Hsapiens.NCBI.GRCh38 has been
>>         available
>>         for one week. But the single-sequence.fa.gz file in this package
>>         is too
>>         big. The sequences of all the chromosomes are put together. It is
>>         difficult to load.  Why do you remake this
>>         package as BSgenome.Hsapiens.UCSC.hg19 do? In
>>         BSgenome.Hsapiens.UCSC.hg19, sequence files for each chromosome
>> are
>>         separated.
>>
>>
>>     Yeah I know... sigh!!  ;-)
>>
>>     All the BSgenome data packages use this new on disk storage format,
>> not
>>     just GRCh38. All the chromosomes are now put together in a single
>>     FASTA file compressed in the RAZip format (extension is rz, not gz).
>>     The file is indexed so it makes direct random access faster. So for
>>     example, extracting only some small portions of the genome with
>>     getSeq() is much faster than with the previous on disk storage format
>>     where the chromosomes were serialized into individual files.
>>     Some people have manifested interest in having getSeq() do fast
>>     random access to the genome sequences for a while. See for example
>>     this post from Michael in 2011:
>>
>>     https://stat.ethz.ch/__pipermail/bioc-devel/2011-May/__002601.html
>>
>>     <https://stat.ethz.ch/pipermail/bioc-devel/2011-May/002601.html>
>>
>>     I'm aware that the new on disk storage format slows down operations
>>     where you actually need to compute on the entire genome, which is
>>     typically done in a loop where one chromosome is loaded at a time.
>>     It's hard to beat the speed (and simplicity) of just loading the
>>     individual serialized DNAString objects in that case. This is
>>     unfortunate and I was a little bit worried about this when I pushed
>>     the new BSgenome packages to the public repos because I think this
>>     is a common use case.
>>
>>     Note that the switch to this new format happened in devel in January
>>     and was announced on the Bioc-devel mailing list:
>>
>>     https://stat.ethz.ch/__pipermail/bioc-devel/2014-__
>> January/005150.html
>>
>>     <https://stat.ethz.ch/pipermail/bioc-devel/2014-January/005150.html>
>>
>>     I was hoping people would test this and maybe start a discussion about
>>     whether this change was worth it or not.
>>
>>     The good news is that I think we can have the best of both worlds i.e.
>>     fast direct random access and fast loading of the full sequences.
>>     I did some testing with the 2bit format and it looks very promising:
>>
>>        - Random access is much faster than with current RAZip'ed FASTA
>>          format,
>>
>>        - Loading a full chromosome into memory is also very fast because
>>          there isn't the overhead of decompressing the data. It could
>>          even be that it's faster than loading the chromosome stored in
>>          a serialized DNAString object, or maybe not, but at least it
>>          should be very close.
>>
>>        - The sequences take less space on disk and the resulting BSgenome
>>          package will also be slightly smaller.
>>
>>     It's something that I was planning to work on early in this new devel
>>     cycle. Seems like I should start even earlier and maybe backport to
>>     BioC release?
>>
>>     Thanks,
>>     H.
>>
>>
>>         Best,
>>         Sean
>>
>>         2014-04-16
>>         ------------------------------__----------------------------
>> --__------------
>>
>>
>>         ____________________________________________________________
>> ________________
>>
>>         devteam-bioc mailing list
>>         To unsubscribe from this mailing list send a blank email to
>>         devteam-bioc-leave@lists.__fhcrc.org
>>         <mailto:devteam-bioc-leave@lists.fhcrc.org>
>>
>>         You can also unsubscribe or change your personal options at
>>         https://lists.fhcrc.org/__mailman/listinfo/devteam-bioc
>>
>>         <https://lists.fhcrc.org/mailman/listinfo/devteam-bioc>
>>
>>
>>     --
>>     Hervé Pagès
>>
>>     Program in Computational Biology
>>     Division of Public Health Sciences
>>     Fred Hutchinson Cancer Research Center
>>     1100 Fairview Ave. N, M1-B514
>>     P.O. Box 19024
>>     Seattle, WA 98109-1024
>>
>>     E-mail: hpages@fhcrc.org <mailto:hpages@fhcrc.org>
>>     Phone: (206) 667-5791 <tel:%28206%29%20667-5791>
>>     Fax: (206) 667-1319 <tel:%28206%29%20667-1319>
>>
>>     _________________________________________________
>>     Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org> mailing
>> list
>>     https://stat.ethz.ch/mailman/__listinfo/bioc-devel
>>     <https://stat.ethz.ch/mailman/listinfo/bioc-devel>
>>
>>
>>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpages@fhcrc.org
> Phone:  (206) 667-5791
> Fax:    (206) 667-1319
>

	[[alternative HTML version deleted]]

