[R-SIG-Finance] [R-sig-finance] Preprocessing RData file (data.table and ff, bigmemory)
jeff.a.ryan at gmail.com
Fri May 22 16:33:10 CEST 2009
>From a recent post by the author:
Further information on 'ff' and 'bigmemory' is covered in those
As far as combining the two/three, I would wait to hear back from Matt
on exactly how to do that. I thought there was an example somewhere
if I recall...
The main advantage to using large datasets in RAM is simply
efficiency. 'ff' makes that process manageable without a lot of RAM,
bigmemory can bypass single process limits of R (and do some cool
memory sharing). The advantage to both is really confined to 32bit
processing, if I am thinking straight.
This is probably more of a question for R-help at this point, or even
R-Sig-db though, as the 'finance' part is only tangential.
If you can break the data up with something like a db scheme, then xts
will be faster than all (?) the other solutions for in-memory
manipulation -- as it is time-series oriented. And if you've got
64bits and lots of RAM it should do most of what you need.
On Fri, May 22, 2009 at 9:18 AM, Steve Jaffe <sjaffe at riskspan.com> wrote:
> I'm new to R and interested in working with large amounts of data
> (timeseries, but regularly spaced.) Can you point me to a good reference for
> using data.table with bigmemory or ff?
> (I'm a bit puzzled about what exactly these packages provide. As I
> understand it, on 32-bit platforms files are subject to the same 2GB limit
> as in-process memory, so I assume that dealing with a larger dataset still
> requires breaking it up into multiple files...)
> Thanks for your help.
> I failed to point out that data.table can make use of both (?) those
> packages. [ff, bigmemory]
> It isn't a time-series library per se, but it make one very cool
> in-memory database. Similar in spirit to some of the not-so-free ones
> out there...
> View this message in context: http://www.nabble.com/high-frequency-data-analysis-in-R-tp23654793p23671313.html
> Sent from the Rmetrics mailing list archive at Nabble.com.
> R-SIG-Finance at stat.math.ethz.ch mailing list
> -- Subscriber-posting only.
> -- If you want to post, subscribe first.
jeffrey.ryan at insightalgo.com
ia: insight algorithmics
More information about the R-SIG-Finance