[R-SIG-Finance] How to input large datasets into R

[Carlos J. Gil Bellosta]

Dear Sarbo,

10000 x 10 and you start having problems? Maybe on a low end netbook.
I have successfully worked with 300,000 x 50 datasets on a regular
desktop PC in the past.

You may also keep your data in a database and access from R. Even if
you do not face serious data size issues, it is good policy.

Packages such as RSQLite or colbycol could also be helpful.

Best regards,

Carlos J. Gil Bellosta
http://www.datanalytics.com


2010/6/29 Sarbo <cmdr_rogue at hotmail.com>:
> The "birch" and "speedglm" packages may be useful. If you have access to
> a multi-core computer with additional memory you may also want to
> download the "REvolution" suite.
>
> Keep in mind that R is not designed to handle giant data sets (though
> having written that, I've processed data sets of dimensions on the order
> of 10000x10 in R before without too many issues). If you really want to
> run robust statistical analyses on large data sets you'll need to go for
> a much more robust language like SAS.
>
> On Tue, 2010-06-29 at 01:51 -0400, Aaditya Nanduri wrote:
>
>> Hello All.
>>
>> For my HW assignment, I was given 30 stocks with minute data (date,
>> time, open, close, high, low, vol) over 7 years.
>>
>> So, each stock has about 610000 rows of data which makes it impossible to
>> calculate z-scores for mean-reversion strategies (required for HW) for even
>> one stock.
>>
>> Is there any way R can read only certain lines of data?
>>
>> For example, in the OU process we use increments of 60. So can R read 1:60,
>> then 2:61 and so on?
>>
>> I recently tried a simple regression on half the data (training set) on my
>> school's computer only to watch it eat up the entire memory leaving me no
>> option but to restart the computer.
>>
>> The data is in .csv format if it matters.
>>
>> Im an undergrad learning about the basic methods in stat arb in an informal
>> setting so you may assume I have absolutely no clue about pretty much
>> anything and everything.
>>
>> And are there any tutorials online for using quantmod? That would be very
>> helpful.
>>
>> Thank you very much.
>>
>> Sincerely,
>> Aaditya Nanduri
>>
>>       [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-SIG-Finance at stat.math.ethz.ch mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> -- Subscriber-posting only. If you want to post, subscribe first.
>> -- Also note that this is not the r-help list where general R questions should go.
>>
>
>
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-SIG-Finance at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.
>