[R-sig-Geo] Memory issue

[Roger Bivand]

On Mon, 3 Nov 2008, Mikael Carlsson wrote:

> 
> Hi,
> 
>  
> 
> First, thanks for all help so far...but I still have some problems.
> Here is the code:
> cumark <- read.table("k:/test/helamarkcu.txt", header=T, sep = ";")

One copy of 3*40K*8 = 1MB

> 
> memory.limit(size = 1897)   
> attach(cumark)
> vew <- as.vector(ew)
> vns <- as.vector(ns)
> vcu <- as.vector(cu)

A second copy

> 
> fcumark <- data.frame(vew, vns, vcu)
>

A third copy (but still of 1MB)

> coordinates(fcumark) <- ~vew+vns 
> 
> L1 <-round((max(vew)- min(vew))/100,0)
> L2 <-round((max(vns)- min(vns))/100,0)
>

So this is a 100m grid? What are L1 and L2? Why not use GridTopology to 
compute this? How many grid cells are you using?

> 
> fcumark.interp.old <- interp.old(vew, vns, vcu,
>                 xo= seq(min(vew), max(vew), length = L1),
>                 yo= seq(min(vns), max(vns), length = L2),
>                 ncp = 0, 
>                 extrap= FALSE,
>                 duplicate = "mean",
>                 dupfun = NULL)
> Then I get this;
> Error: cannot allocate vector of size 351.8 Mb
> 
> When I tested with a smaller file (~2000 rows) I come to the end without
> problems.

With the same L1, L2 (possibly not)? They must be the main problem, no? As 
suggested, you could tile the interpolations and merge them afterwards. 
The main thing is the size of L1*L2.

> 
> Rest of the code..
> 
> fcumark.interp.old.grid <-
> as.SpatialGridDataFrame.im(as.im(fcumark.interp.old))

Lots more copying

> 
> ##  spplot(fcumark.interp.old.grid, col.regions=bpy.colors(),
> sp.layout=list("sp.points", pch="+", fcumark))
> 
> slot(slot(fcumark.interp.old.grid, "grid"), "cellsize") <-
> rep(mean(slot(slot(fcumark.interp.old.grid,
>    "grid"), "cellsize")), 2)
>

Messy - use a Grid Topology as input.

Roger

> write.asciigrid(fcumark.interp.old.grid[1], "k:/test/helamarkcu.asc",
> na.value = -9999)
> ## End run
>
> helamarkcu.txt is a plain ascii file, looks like...
> ew;ns;cu
> 1344628;6138662;24.0
> 1340104;6138880;17.0
> 1342246;6138946;21.0
> 1347259;6140911;19.0
> 1339968;6140932;15.0
> 1349906;6140987;21.0
> ...
> ..
> .
> (38155 rows)
> Have anybody a solution? Writing result from interp.old direct to
> disk/file on the "fly"?
> 
> Running R 2.7.2 and WinXP
> 
> Regards,
> Mikael
> 
> 
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no