On Fri, May 13, 2011 at 10:36 AM, Marc Carlson <mcarlson@fhcrc.org> wrote:

>  On 05/12/2011 09:42 PM, Michael Lawrence wrote:
>
>
>
> On Thu, May 12, 2011 at 10:35 AM, Marc Carlson <mcarlson@fhcrc.org> wrote:
>
>> Hi Michael,
>>
>> That is an interesting idea.  I like the idea of having more data be
>> available via FeatureDb, and I especially like the idea of having useful
>> transformations of the data it provides.  But I am a little confused about
>> one part of what you are suggesting.  Is there a reason why we would want to
>> add a bunch of stuff to basically re-implement what the database does
>> instead of just writing some simple methods to allow the import of these
>> other kinds to files?
>>
>> I can think of advantages to keeping the data container type consistent
>> (providing that it is not proving burdensome), since SQL allows joins to be
>> made across databases thus allowing a collection of data that has all been
>> stored in this way to be easily linked together as needed.  But what is the
>> advantage of making a bunch of classes and methods that will allow us to
>> pretend that our bam and vcf files are actually databases?
>>
>>
> I don't want to pretend that they are first-class databases. It would just
> be nice to have a common interface around these various data sources. The
> simplest interface would allow range-based (GRanges) queries and return a
> GRanges as the result. Each data source will have its own specific
> parameters, but those could be fields of the particular Db class and held
> constant across multiple range queries.
>
> The driving use-case for this is visualization. The user is looking at a
> particular region of a track. That track has an underlying data source,
> often on disk. We just need to query for the features in that region.
>
>
> Hi Michael,
>
> In that case, we will have to write something that can match the features
> in format "x" to a GRanges object regardless of implementation.  So wouldn't
> it be simpler to just write a single set of import and display functions?
> IOW, why not just write a simple set of import methods so that we can import
> data from gff, or bam or vcf etc such that they all end up in the same basic
> container?  I understand that we could achieve the same end result by also
> using a more complex class structure, but what is the upside to adding that
> additional complexity?
>

While having an import() method on each data structure could work for simple
cases, we want to be able to treat these data sources equivalently, to the
extent that they all inherit from some common base class. The motivator here
is a graphics pipeline, which is essentially a chain of data models that
pull/push information from/to the previous model and transform, subset or
otherwise compute on it.

My current thinking has led me away from a special RangeDb object and
towards abstracting things at the GRanges-level. GRanges is already nicely
abstracted from its S4 slot implementation through the abstract base class
GenomicRanges. We've used this to implement a range container based on
reference classes. It wouldn't be too hard, with some caching for
performance, to implement a "proxy" GenomicRanges that can pull from some
other GenomicRanges and do something to the data on the way out.

We can just implement GenomicRanges against your FeatureDb if we wanted
features stored in SQLite. Thus, this work really doesn't affect
GenomicFeatures anymore.

I think the upside to using a single container type on the back end is that
> these containers can all be accessed under the hood in a similar manner,
> joined together as needed and even indexed when appropriate.  And I worry
> that if we take the more complicated approach and then someone wants to do
> operations like this they may have to worry about "reinventing the wheel"
> 1st before they can proceed.  With a single container approach, it seems
> that we get the benefits of previous generations of database features plus
> whatever other operations we also enable with our GRanges infrastructure on
> top.  But with the more complex design we are forced to only use whatever
> things we have re-implemented in R.  If for some reason there are unforeseen
> bottlenecks, I worry that we might have less flexibility with the design you
> are proposing than if we just did something simpler.
>
> This "two layers of access" approach has already come in handy on several
> occasions in the GenomicFeatures package and it was convenient on those
> occasions to be able to choose the method (direct database access vs access
> as a GRanges object) that gave better performance.  It was nice to have a
> couple of alternative access options to choose from.  And I will grant you
> that so far, in most of those cases the operations have been very simpls and
> so the quickest option was to actually to just use the very fast GRanges
> operations.  ;)  But as workflows get more complicated, it is easy to
> imagine that this might not always be the case.  And indeed the future will
> probably eventually involve bringing together many different kinds of these
> data resources together in compound ways and so that starts to look more and
> more like a job for a traditional relational database.  It might be nice to
> be able to leverage that when the time comes.
>
> Curious to hear what you think about this.
>
>
>
>
  Marc
>
>
>
>
>
>> Also what would the purpose of a SequenceDb object be?  The name is
>> generic enough that I am unable to guess what you have in mind.
>>
>>
> A data source for sequences. Could be implemented with a BSgenome object,
> or FA indexed file, etc.
>
>
>>  Marc
>>
>>
>>
>>
>> On 05/12/2011 06:08 AM, Michael Lawrence wrote:
>>
>>>  Hi guys,
>>>
>>> I was just looking at the FeatureDb class in GenomicFeatures. I'm
>>> wondering
>>> if we couldn't abstract that from its SQLite implementation. There are
>>> many
>>> other sources of features, e.g., files like BAM, VCF and even BED. If
>>> these
>>> are indexed properly, we could make fast queries against them. So what we
>>> really need is a class, named something like FeatureDb, that returns, for
>>> a
>>> given 'which' (as a bare minimum), a GRanges.
>>>
>>> I could also imagine having proxy FeatureDb objects that transform the
>>> data
>>> on the way. Like a FeatureDb that will return the coverage, using another
>>> FeatureDb as a source. Caching could be implemented as part of the base
>>> class. I'm also wondering whether these should be reference classes. Then
>>> if
>>> some "parent" FeatureDb is modified, the down-stream objects can be
>>> informed
>>> of the change.
>>>
>>> And a SequenceDb would be nice, too.
>>>
>>> I'll write up a prototype in the MutableRanges package (in the bioc
>>> repo),
>>> but I'll call it RangeDb to avoid conflicts for now.
>>>
>>> Michael
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> Bioc-devel@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>>
>>
>> _______________________________________________
>> Bioc-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
>
>

	[[alternative HTML version deleted]]

