[R-sig-Geo] Large dataset

[Roger Bivand]

On Wed, 11 Jul 2007, KAM Tin Seong wrote:

> Hi all, I have a large dataset of 360,000 sample points.  When I perform
> analysis using Khat function of splancs, the following error messages
> prompted:
>
> Error: cannot allocate vector of size 589.6 Mb
> In addition: Warning messages:
> 1: Reached total allocation of 1013Mb: see help(memory.size)
>
> It would be very much appreciated if someone could advice me how to
> overcome this problem.

The obvious first question is why? If you take a sample of 10000 points, 
you will get a very good approximation of the khat value at the s-bins - 
so why try all 360K? Either take tiles of your data set, or some other 
partition - since you are assuming anyway that the process is the same 
over the whole area, aren't you? You will certainly find that simulating 
from a CSR for 360K to put an envelope on the result will also be memory 
intensive.

By the way, always include your specific code, and the output of 
sessionInfo(). It may be that you have a very complex polygon windoe, 
and/or too much detail in the s-bins. Do you get the same memory 
constraint in spatstat, or equivalently in spatial?

I assume that you are looking to see if the point pattern is clustered, is 
that correct? If you really need to use all the points, we'd need to debug 
the command to see where the memory allocation is occurring, since it may 
be an avoidable copying operation somewhere. More RAM is another option.

Roger

> Dr Kam Tin Seong
> Practice Associate Professor
> School of Information Systems
>
> Singapore Management University
> 80 Stamford Road
> Singapore 178902
> e-mail: tskam at smu.edu.sg
> DID: + 65 6828 0932
>
>
>
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no