Dear Stefan,

Thanks a lot for the tip. This is definitely starting to sound closer to what I need. Best,

Greg.

Gregory Jefferis

On 21 Nov 2012, at 23:06, Stefan Theussl <stefan.theussl@gmail.com> wrote:

> Hi Gregory,
> 
> perhaps package DSL offers what you have in mind. It provides an abstract distributed list class. Corresponding objects are list-type objects which use a distributed storage to store their elements. By default elements (arbitrary R objects) are stored on the local disk. Only those elements which are needed for computation are loaded. Parallelization is supported via the parallel package. What is more, you have the option to scale out using Hadoop. Please check the package vignette for more details [1].
> 
> Best,
> Stefan
> 
> [1] http://cran.r-project.org/web/packages/DSL/index.html
> 
> 
> On Mon, Nov 19, 2012 at 8:57 AM, Gregory Jefferis <jefferis@gmail.com> wrote:
> Dear Saptarshi,
> 
> Thanks for your response.
> 
> On 17 Nov 2012, at 05:42, Saptarshi Guha wrote:
> 
> > One question: how large is each list element? < 256MB?
> 
> fairly small (<5Mb)
> 
> > One approach: store your data in Hbase (or Hadoop MapFiles), with key
> > as the list index.
> 
> Can Hbase cope with arbitrary format data (like a list) rather than just tables?
> >
> > The define an object O of class C. Redefine "[[".C as a function that
> > reads HBase/HDFS queries for the list index i (e.g. as in x[[i]]) and
> > retrieve the i'th list element.
> 
> Yes, that was the sort of thing I had in mind.
> 
> > Cache this, so the second x[[i]] is
> > called, it will retreive it from the cache. To prevent the cache from
> > expanding to 2GB, you can keep last K cache entries (some MRU/LRU type
> > cache retention scheme).
> 
> I was hoping someone might have done some work on this type of strategy that could be reused!
> 
> > Not sure how you intend to use this list, the application above
> > handles the query of some subset of keys (indices 1...16K).
> > Do you want to run some function F across all/large subset of keys?
> > This is a good case for R and Hadoop.
> 
> I have 2 use cases – one batch processing on a cluster where all elements will be processed, typically in a function that takes a pair of elements as inputs and computes scores for all 16k^2 combinations; this is much faster when. For the time being simple approaches seem to work here, but Hadoop is something I have been meaning to investigate for the longer term.  The second main use case, interactive use on single user machine, where I am more concerned about saturating memory, is what I am trying to address currently.
> 
> Best wishes,
> 
> Greg.
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> 
> 

	[[alternative HTML version deleted]]

