[Rd] arbitrary size data frame or other stcucts, curious about issues invovled.
marchywka at hotmail.com
Sat Jun 11 14:42:03 CEST 2011
We keep getting questions on r-help about memory limits and
I was curious to know what issues are involved in making
common classes like dataframe work with disk and intelligent
swapping? That is, sure you can always rely on OS for VM
but in theory it should be possible to make a data structure
that somehow knows what pieces you will access next and
can keep thos somewhere fast. Now of course algorithms
"should" act locally and be block oriented but in any case
could communicate with data structures on upcoming
access patterns, see a few ms into the future and have the
right stuff prefetched.
I think things like "bigmemory" exist but perhaps one
issue was that this could not just drop in for data.frame
or does it already solve all the problems?
Is memory management just a non-issue or is there something
that needs to be done to make large data structures work well?
marchywka at phluant.com
Online Advertising and Analytics for Mobile
More information about the R-devel