[R-sig-Geo] gstat memory

[Edzer Pebesma]

Dear Murray and list,

repairing the memory leaks turned out to be a more fun job than I
thought it would be. Memory leakage mainly took place in the
neighbourhood search algorithm (nsearch.c) which uses a quadtree for 2
and octtree (octree?) for 3-dimensional data. As you can guess, the code
is full with recursion, and was contributed by Steve Joyce, someone much
cleverder than I am, back in 1997, so that was a nice puzzle. But the
leaks should be gone now.

gstat_0.9-64 now propagates through CRAN.

In my enthousiasm, I removed any other memory leak I could. This meant
several code changes at the c level that looked as if back then I made
them to resolve a bug -- of course we didn't have valgrind then. In
other words, I might have introduced new problems. If you encounter
anything unexpected, then please let me know.

Whenever this stabilizes, I will start calling increase gstat's version
number to 1.0_0

Best regards,
--
Edzer


Murray Richardson wrote:
> Hello R-SIG-GEO list,
>
> I know this has come up before but I am having an ongoing memory
> problem with the gstat package (gstat out of dynamic memory) that I
> can't seem to solve.
>
> I am using R to interpolate DEMs from LiDAR xyz point files and mosaic
> them together via RSAGA.  The script uses a loop to load in each xyz
> point file, and interpolate over a regular lattice of points using idw
> from gstat.  Although large, the computational requirements within
> each iteration of the loop should be well within my system's ability
> so it seems like it is a cumulative effect (note I can restart the
> process at the last loop that triggered the error and the iteration is
> successful).  I am removing temp objects and running gc() at the end
> of every loop.
>
> It proceeds normally for about 5-10 iterations and physical memory use
> on my system (VISTA 64 bit, 12GB RAM) gradually increases over time
> until I get the "gstat out of memory" error. Here is the relevant
> portion of the loop:
>
> ...
>
>   path<-"E:/LidarData/1_Ground_First_Return/UTM17/"
>   filen<-paste(path, centre_tile$TILECODE, ".xyz", sep="")
>   xyzi<-read.table(filen, sep=",", header=F)
>   names(xyzi)<-c("x","y","z","i")
>
>   for(j in 1:length(tile_group_names)){
>       filen<-paste(path, tile_group_names[j], ".xyz", sep="")
>       tmp_xyzi<-read.table(filen, sep=",", header=F)
>       names(tmp_xyzi)<-c("x","y","z","i")
>       xyzi<-rbind(xyzi,tmp_xyzi)
>
>   }
>       coordinates(xyzi)=~x+y
>   # note grid_coords is just the regular lattice that is created from
> the current tile coordinates
>   coordinates(grid_coords)=~x+y
>
>   # do the interpolation
>   interp<-idw(z~1, xyzi, grid_coords, nmax=4,maxdist=2, idp=1.0)
>
>   names(interp)<-c("z","var")
>   slot(interp,"data")<-data.frame(slot(interp,"data"))
>   finalSPntsDF<-SpatialPointsDataFrame(interp, data.frame(interp$z),
>               proj4string = CRS(as.character(NA)), match.ID = TRUE)
>
>
>   finalSPDF<-SpatialPixelsDataFrame(finalSPntsDF,
> data.frame(interp$z), tolerance = sqrt(.Machine$double.eps),
>     proj4string = CRS(as.character(NA)))
>
> ... On a related note - I have tried using the 64 bit version of
> Revolution but unfortunately the gstat package has not been ported. 
> Has anyone contemplated or begun a 64 bit port of this package?
>
> Thanks
>
> Murray
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

-- 
Edzer Pebesma
Institute for Geoinformatics (ifgi), University of Münster
Weseler Straße 253, 48151 Münster, Germany. Phone: +49 251
8333081, Fax: +49 251 8339763 http://ifgi.uni-muenster.de/
http://www.springer.com/978-0-387-78170-9 e.pebesma at wwu.de