Stella,

A few brief words of advice:

1. Work through your code a line at a time, making sure that each is what
you would expect.  I think some of your later problems are a result of
something
early not being as expected.  For example, if the read.delim() is in fact
not
giving you what you expect, stop there before moving onwards.  I suspect
some funny character(s) or character encodings might be a problem.

2. 32-bit Windows can be limiting. With 2 GB of RAM, you're probably not
going to be able to work effectively in native R with objects over 200-300
MB,
and the error indicates that something (you or a package you're using)
simply
have run out of memory.  So...

3. Consider more RAM (and preferably with 64-bit R).  Other solutions might
be possible, such as using a database to hand the data transition into R.
2.5 million rows by 18 columns is apt to be around 360 MB.  Although you
can afford 1 (or a few) copies of this, it doesn't leave you much room for
the memory overhead of working with such an object.

Part of the oringal message below.

Jay

-------------------------------------------------------------

Message: 80
Date: Mon, 19 Apr 2010 22:07:03 +0200
From: Stella Pachidi <stella.pachidi@gmail.com>
To: r-help@stat.math.ethz.ch
Subject: [R]  Huge data sets and RAM problems
Message-ID:
       <g2j133363581004191307t2a48c1bfqd9d57cf0d6c629a6@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Dear all,

....

I am using R 2.10.1 in a laptop with Windows 7 - 32bit system, 2GB RAM
and CPU Intel Core Duo 2GHz.

.....

Finally, another problem I have is when I perform association mining
on the data set using the package arules: I turn the data frame into
transactions table and then run the apriori algorithm. When I put too
low support in order to manage to find the rules I need, the vector of
rules becomes too big and I get problems with the memory such as:
Error: cannot allocate vector of size 923.1 Mb
In addition: Warning messages:
1: In items(x) : Reached total allocation of 153Mb: see help(memory.size)

Could you please help me with how I could allocate more RAM? Or, do
you think there is a way to process the data by loading them into a
document instead of loading all into RAM? Do you know how I could
manage to read all my data set?

I would really appreciate your help.

Kind regards,
Stella Pachidi


-- 
John W. Emerson (Jay)
Associate Professor of Statistics
Department of Statistics
Yale University
http://www.stat.yale.edu/~jay

	[[alternative HTML version deleted]]

