[R] amount of data R can handle in a single file

[Prof Brian Ripley]

On Thu, 17 Feb 2011, Nasila, Mark wrote:

> Dear Sir/Madam,
>
>
>
>  I would like to know what is the maximum number of observations a
> single file must have when using R. I am asking this because am trying
> to do research on banking transactions and i have around 49million
> records. Can R handle this? Advise with regard to this.

Depends on the platform and how many fields there are in a record. 
(On a 64-bit platform we have handled databases of 70m records and 
about 30 fields: we did use a DBMS to store them, though: see the 'R 
Data Import/Export Manual'.)

OTOH, one could ask what extra useful information there is in 49m 
records over a 1% sample.  (In our case it was rare combinations, and 
we simply extracted those separately from the DBMS.)

> Mark Nasila
> Quantitative Analyst
> CBS Risk Management
>
> Personal Banking
> 7th Floor, 2 First Place,
> Cnr Jeppe and Simmonds Street,
> Johannesburg,
> 2000
> Tel (011) 371-2406, Fax (011) 352-9812, Cell 083 317 0118
> e-mail MNasila at fnb.co.za <mailto:MNasila at fnb.co.za>
>
> www.fnb.co.za <http://www.fnb.co.za/>  www.howcanwehelpyou.co.za
> <http://www.howcanwehelpyou.co.za/>
>
> First National Bank - a division of FirstRand Bank Limited.
> An Authorised Financial Services and Credit Provider (NCRCP20).
>
> 'Consider the effect on the environment before printing this email.'
>
>
>
>
> To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
> https://www.fnb.co.za/disclaimer.html
>
> If you are unable to access the Disclaimer, send a blank e-mail to
> firstrandbankdisclaimer at fnb.co.za and we will send you a copy of the Disclaimer.
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595