[R] memory, i am getting mad in reading climate data

[jim holtman]

Another suggestion is to start with a subset of the data file to see
how much memory is required for your processing.  One of the
misconceptions is that "memory is free".  People think that with
virtual memory and other such tools, that there is no restriction on
what you can do.  Instead of starting off, and getting "mad", when
trying to read in all your data, try a small subset and then look at
the memory usage of the R process.  I would assume that you are
running on a 32-bit Windows system, which if you are lucky, can
address 3GB for a user process.  My rule of thumb is that the largest
object I can work with is 30% of the real memory I have available, so
for my Windows system which lets me address almost 3GB, the biggest
object I would attempt to work with is 1GB,

Working with a subset, you would understand how much memory an XXXMB
file might require.  This would then give you an idea of what the
maximum size file you might be able to process.

Every system has limits.  If you have lots of money, then invest in a
64-bit system with 100GB of real memory and you probably won't hit its
limits for a while.  Otherwise, look at taking incremental steps and
possibly determining if you can partition the data.  You might
consider a relational database to sotre the data so that it is easier
to select a subset of data to process.



2012/3/17 Uwe Ligges <ligges at statistik.tu-dortmund.de>:
>
>
> On 17.03.2012 19:27, David Winsemius wrote:
>>
>>
>> On Mar 17, 2012, at 10:33 AM, Amen wrote:
>>
>>> I faced this problem when typing:
>>>
>>> temperature <- get.var.ncdf( ex.nc, 'Temperature' )
>>>
>>> *unable to allocate a vector of size 2.8 GB*
>>
>>
>> Read the R-Win-FAQ
>
>>
>>>
>>>
>>> By the way my computer memory is 4G and the original size of the file is
>>> 1.4G,netcdf file
>
>
> ... and reading / storing the data in memory may require much more than
> 4GB...
>
> Uwe Ligges
>
>
>>> I don't know what is the problem.Any suggestion please
>>> I tried also
>>> memory limit(4000)
>>> 4000
>>> but didnt solve the problem.any help
>>
>>
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.



-- 
Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.