[R-SIG-Finance] [R-sig-finance] Preprocessing RData file (data.table and ff, bigmemory)

[Jeff Ryan]

>From a recent post by the author:

http://finzi.psych.upenn.edu/R/Rhelp08/2009-March/193490.html

Further information on 'ff' and 'bigmemory' is covered in those
respective packages.

As far as combining the two/three, I would wait to hear back from Matt
on exactly how to do that.  I thought there was an example somewhere
if I recall...

The main advantage to using large datasets in RAM is simply
efficiency.  'ff' makes that process manageable without a lot of RAM,
bigmemory can bypass single process limits of R (and do some cool
memory sharing).  The advantage to both is really confined to 32bit
processing, if I am thinking straight.

This is probably more of a question for R-help at this point, or even
R-Sig-db though, as the 'finance' part is only tangential.

If you can break the data up with something like a db scheme, then xts
will be faster than all (?) the other solutions for in-memory
manipulation -- as it is time-series oriented.  And if you've got
64bits and lots of RAM it should do most of what you need.

HTH
Jeff


On Fri, May 22, 2009 at 9:18 AM, Steve Jaffe <sjaffe at riskspan.com> wrote:
>
> I'm new to R and interested in working with large amounts of data
> (timeseries, but regularly spaced.) Can you point me to a good reference for
> using data.table with bigmemory or ff?
>
> (I'm a bit puzzled about what exactly these packages provide. As I
> understand it, on 32-bit platforms files are subject to the same 2GB limit
> as in-process memory, so I assume that dealing with a larger dataset still
> requires breaking it up into multiple files...)
>
> Thanks for your help.
>
>
>
> I failed to point out that data.table can make use of both (?) those
> packages. [ff, bigmemory]
>
> It isn't a time-series library per se, but it make one very cool
> in-memory database.  Similar in spirit to some of the not-so-free ones
> out there...
>
> Jeff
>
>
> --
> View this message in context: http://www.nabble.com/high-frequency-data-analysis-in-R-tp23654793p23671313.html
> Sent from the Rmetrics mailing list archive at Nabble.com.
>
> _______________________________________________
> R-SIG-Finance at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only.
> -- If you want to post, subscribe first.
>



-- 
Jeffrey Ryan
jeffrey.ryan at insightalgo.com

ia: insight algorithmics
www.insightalgo.com