[R-sig-eco] Using ME() of spdep: too many observations? how to make it step by step?

[Carsten Dormann]

Dear nameless user(ess) (vega_l),

as you correctly point out, the data set is "kind of large". ME requires 
the inversion of the n x n matrix, which is a computer-intensive task 
for a 4000 x 4000 matrix.
I propose you could do two things:
1. check if something is wrong with the code/data in principle, by using 
only a random subset of, say, 300 data points.
2. I extracted the following code from the ME function and put it into a 
function:

getSEVMs <- function(listw){
  listw <- listw2U(listw) # make weights symmetric
     Wmat <- listw2mat(listw) # convert to full matrix form
    n <- ncol(Wmat)
  Cent <- diag(n) - (matrix(1, n, n)/n)
  eV <- eigen(Cent %*% Wmat %*% Cent, EISPACK=TRUE)$vectors
  eV
}
Use this to find out whether R is able to correctly calculate the 
spatial EVs of this data set. ME repeatedly calls this bit of code (I 
seem to remember), and it does the key step of calculating the EVs.

If both 1. and 2. work, I recommend giving R more memory (see 
?memory.limit), which will slow it down because now it uses part of the 
hard drive as RAM, and/or using a different computer (preferably a 
64-bit system that can manage larger RAM than "only" 3 and a bit GB). I 
am lucky enough to have access to a Linux-cluster and even there your 
sample size is about the limit of what is feasible.

Note that it would be very useful to know which error R returned when it 
"halted"! Was it a memory overflow ("can't allocate a vector of ... MB") 
or was it a functional error ("Error in eigen(.)")? The latter indicates 
that your data are too many, the former that you want a bigger computer 
memory.

HTH,

Carsten



vega_l at yahoo.com wrote:
> Good afternoon and greetings to everyone!
>
> We are trying to remove the spatial structure of our data by employing Moran Eigen Vectors with the spdep package. Our dataset is kind of large, 3694 samples and 13 variables.
>
> The bad news: my Dual Core 2.0 MHz laptop with Ubuntu 9.04 (2.0 Gb RAM) halted after almost 4 days of continuous processing, and all the effort was lost. (Why the calculations took so much time??)
>
> I wonder if there is a way to accelerate the calculations with ME() and to save the resulting eigenvalues as they get calculated. Or if it is possible to make the calculation process of ME() step by step.
>
> The code in R is a follows:
>
> taimin.xy = taimin[c("dy", "dx")];
> coordinates(taimin.xy) <- ~dy+dx; #dy, dx, name of the respective
> coordinate columns
> coords<-coordinates(taimin.xy);
> library(spdep);
> TaiminGabrielGraph<-gabrielneigh(coords, nnmult = 12);
> tai.gab.nb<-graph2nb(TaiminGabrielGraph,sym=TRUE);
> nbtaim_distsg <- nbdists(tai.gab.nb, coords);
> nbtaim_simsg <- lapply(nbtaim_distsg, function(x) (1-((x/(4*50))^2)) );
> MEtaig.listw <- nb2listw(tai.gab.nb, glist=nbtaim_simsg, style="B");
> sevmtaig <- ME(Presabs
> ~Age+Curv+Zon2005+ZoneMeiji+Wi+Sol+Slope+Seadist+Elev+Basin,data=taimin,
> family=binomial,listw=MEtaig.listw)
>
> I hope someone can give me some help.
>
> Thank you!
>
>
>       ____________________________________________________________________________________
> ¡Obtén la mejor experiencia en la web!
> Descarga gratis el nuevo Internet Explorer 8. 
> http://downloads.yahoo.com/ieak8/?l=e1
>
> _______________________________________________
> R-sig-ecology mailing list
> R-sig-ecology at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>
>   

-- 
Dr. Carsten F. Dormann
Department of Computational Landscape Ecology
Helmholtz Centre for Environmental Research-UFZ 
Permoserstr. 15
04318 Leipzig
Germany

Tel: ++49(0)341 2351946
Fax: ++49(0)341 2351939
Email: carsten.dormann at ufz.de
internet: http://www.ufz.de/index.php?de=4205