[R] Running out of memory

[Peter Adler]

I'm working with large data frames and running out of memory.  I hope some 
of you may be able to suggest a more efficient approach.

I have grid/lattice data representing a time series of 1 m2 quadrats in a 
grassland: Each 1 cm2 cell or pixel contains one ecological state (ie grass 
or bare ground).  The goal is to calculate, for each cell, the transition 
probabilities to all available states (given that a cell is occupied by 
grass, what are the probabilities it will change to grass or bare ground in 
the next time step?).  I am using multinom (in package nnet, MASS) to 
calculate these transitions as a function of the density of each state in 
some defined neighborhood around the focal cell.  So I generate a data 
frame with the following columns:

quadrat, year, x coordinate, y coordinate, state at time t, state at time 
t+1, density of state 1 at time t, density of state 2,...state n.

Thus each quadrat to quadrat transition, using 100x100 cell quadrats, can 
generate 10,000 records.  Right now I import, make calculations and store 
the data for each quadrat-to-quadrat transition in a temporary array, then 
use rbind to append this array on to the whole (final) data frame, then 
repeat for the next year (re-using the same temporary array).  I use up my 
max memory allocation (1024Mb) after about 130 quadrat-years of data.  I 
could increase my max memory allocation some more, but this will simply 
raise the ceiling, not solve the problem.

I don't understand why R runs out of memory so soon, since text files 
containing this same data are much smaller. For example, 35 years of data 
for one quadrat uses only 11Mb when stored in a text file on my hard drive, 
but when I import it into R it occupies over 100Mb (according to 
memory.size() ).

Should I think about exporting the data to text files as I go?

Thanks for your help,

Peter
------------------------
Peter Adler, PhD
Dept. Ecology, Evolution and Marine Biology
University of California
Santa Barbara, CA 93106
tel: (805) 893-7416
http://www.lifesci.ucsb.edu/~adler/