[Rd] arbitrary size data frame or other stcucts, curious about issues invovled.

[Mike Marchywka]

We keep getting questions on r-help about memory limits  and
I was curious to know what issues are involved in making 
common classes like dataframe work with disk and intelligent
swapping? That is, sure you can always rely on OS for VM 
but in theory it should be possible to make a data structure
that somehow knows what pieces you will access next and
can keep thos somewhere fast. Now of course algorithms 
"should" act locally and be block oriented but in any case
could communicate with data structures on upcoming 
access patterns, see a few ms into the future and have the 
right stuff prefetched.
 
I think things like "bigmemory" exist but perhaps one
issue was that this could not just drop in for data.frame
or does it already solve all the problems?
 
Is memory management just a non-issue or is there something
that needs to be done  to make large data structures work well?
 
 
Thanks.
 
 
-------------------

415-264-8477
marchywka at phluant.com

Online Advertising and Analytics for Mobile
http://www.phluant.com