[Bioc-devel] Update on SQLite-based annotation data package (prototype available)

[Seth Falcon]

Wolfgang Huber <huber at ebi.ac.uk> writes:

> Hi Seth,
>
> I installed the package, but I get:
>
>> ? hgu95av2db
> No documentation for 'hgu95av2db' in specified packages and libraries:
> you could try 'help.search("hgu95av2db")'
>
>> ?getDb
> No documentation for 'getDb' in specified packages and libraries:
> you could try 'help.search("getDb")'
>
>> ?hgu95av2CHRLOC
> No documentation for 'hgu95av2CHRLOC' in specified packages and libraries:
> you could try 'help.search("hgu95av2CHRLOC")'
>
> and there is also no vignette

Yep.  It really is a prototype.  To get started, try pretending you
have called library(hgu95av2).  IOW, you should have all the same
"environments" (in quotes because now they are S4 instances) and can
treat them as such.

We will put some documentation together for the experimental APIs we
are working on, but things are in flux.  Herve has a vignette like
document that we will post asap.

Some notes on performance are worth noting...  The database approach
is going to be slower than having everything in memory for many
operations.  When retrieving annotation for reasonably small gene
lists, the difference is not huge.  However, for operations that pull
everything from a given mapping, such as as.list(), you will see a
huge difference.  

So why are the SQLite-based packages a good thing?  Here are some
thoughts:

  1. They will allow us to deal with much larger data collections.
     The environment-based packages require being able to have all of
     the data in memory at once and provide no easy way to unload the
     data once it has been loaded.  The SQLite-based packages can
     easily handle much larger data sizes and pull only the requested
     data into memory at any one time.

  2. More flexible queries.  With the SQLite-based packages, many
     queries that currently require loops over possible many entire
     environments can be accomplished in one statement.  Using some
     simple SQL statements, I've been able to improve the performance
     of the hyperGTest function by 10x.  Focused queries will
     generally be much faster with the SQLite-based packages.

+ seth







>
>> sessionInfo()
> R version 2.5.0 Under development (unstable) (2007-01-22 r40543)
> x86_64-unknown-linux-gnu
>
> locale:
> LC_CTYPE=it_IT.UTF-8;LC_NUMERIC=C;LC_TIME=it_IT.UTF-8;LC_COLLATE=it_IT.UTF-8;LC_MONETARY=it_IT.UTF-8;LC_MESSAGES=it_IT.UTF-8;LC_PAPER=it_IT.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=it_IT.UTF-8;LC_IDENTIFICATION=C
>
> attached base packages:
> [1] "tools"     "stats"     "graphics"  "grDevices" "utils"     "datasets"
> [7] "methods"   "base"
>
> other attached packages:
>    hgu95av2db AnnotationDbi       RSQLite           DBI       Biobase
>     "1.13.91"      "0.0.41"      "0.4-19"      "0.1-12"     "1.13.34"
>      fortunes
>       "1.3-2"
> Cheers
>  Wolfgang
>
>
>> 
>> We are making progress on converting the annotation data packages to
>> use SQLite as the backend storage mechanism.
>> 
>> The devel annotation package repository has a prototype of a
>> SQLite-based annotation data package (hgu95av2db).  If you are running
>> R-devel, then you should be able to install it via biocLite (sorry,
>> only source package at this point).
>> 
>> The SQLite-based annotation packages depend on the AnnotationDbi
>> package which provides an environment-like interface that should be
>> backwards compatible.  Advanced users can get a connection to the DB
>> and issue raw SQL queries.  We are also planning to provide more
>> convenience/accessor functions along the lines of the annotate
>> package.
>> 
>> Our plan for the upcoming 2.0 release of Bioconductor is to include
>> both environment-based and SQLite-based annotation packages.
>> 
>> If you maintain a package that makes use of annotation data packages,
>> it would be good to see if the hgu95av2db prototype will work with
>> your code (if not, please let us know).
>> 
>> + seth
>> 
>> _______________________________________________
>> Bioc-devel at stat.math.ethz.ch mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
>
> -- 
> ------------------------------------------------------------------
> Wolfgang Huber  EBI/EMBL  Cambridge UK  http://www.ebi.ac.uk/huber