[R-sig-Geo] errorsarlm out of memory (method Chebyshev undetected in help)

[Roger Bivand]

On Sat, 26 Jun 2010, elaine kuo wrote:

> Thanks for reminding.
>
> method "Sparse" and "Chebyshev"" unavailable here

There is no method="Sparse", "Chebyshev" is in recent releases of spdep, 
yours is old.

>
> memry.size ()  TRUE=1535.12  / FALSE=1147.03
> memory.limit() size=NA  1535
>
> 1. spdep  ver 2.10.1
>
> 2. the code is as below.
>
> rm(list=ls())
> datam <-read.csv("c:/migration/M_R_20100618_t2.csv",header=T, row.names=1)
> dim(datam)
> datam[1,]
>
> #SAR err
> library(ncf)
> require (spdep)
>
> # Define coordinates, neighbours, and spatial weights
> coords<-cbind(datam$lat,datam$lon)
> coords<-as.matrix(coords)
>
> #Define neighbourhood (here distance 9)
> nb10<-dnearneigh(coords,0,9)    # 9
>
> #######################
> #Spatial weights, illustrated with coding style "W" (row standardized)
> nb10.w<-nb2listw(nb10, glist=NULL, style="W", zero.policy=FALSE)
>
> #style="C"(sums over all links to n), or"B"(binary) for method "Matrix"or
> "spam" in errorsarlm
> #"C" and "B" = symmetric but "W"(sums over all links to n), "S" = similar to
> symmertic
> #"U"(sums over all links to unity)  , "S"( variance-stabilizing coding )
>
>
> glm1<- glm (datam$bird_w~datam$topo_var)
> summary(glm1)
> res.glm1 <- residuals(glm1)
>
>
> #######################
> #1. Spatial SAR error model
> sem.nb10.w <- errorsarlm(glm1,

*NO* - the first argument to models is a formula, I'm afraid that you need 
to consult local help (from someone who can talk to you where you work) to 
sort out how to use R. This may be trying to extract the formula from 
glm1, but do not rely on this.

Once you start doing things properly (start with a small and simple 
example), you can scale up to harder things.

Roger


> listw=nb10.w,na.action=na.fail,tol.solve=1.0e-12)
> summary(sem.nb1.5.w) #Gives a summary of the SAR error model
> res.sem.nb1.5.w <- residuals(sem.nb1.5.w)
> #######################
>
>
>
>
> 3. error message
> method=default
> warnings
> cannot allocate vector of size 181.2 Mb
>> warnings()
> Warning messages:
> 1: In errorsarlm(glm1, listw = nb10.w, na.action = na.fail,  ... :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 2: In errorsarlm(glm1, listw = nb10.w, na.action = na.fail,  ... :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 3: In crossprod(WA) : Reached total allocation of 1535Mb: see
> help(memory.size)
> 4: In crossprod(WA) : Reached total allocation of 1535Mb: see
> help(memory.size)
> 5: In array(0, c(n, p)) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 6: In array(0, c(n, p)) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 7: In t.default(W) : Reached total allocation of 1535Mb: see
> help(memory.size)
> 8: In t.default(W) : Reached total allocation of 1535Mb: see
> help(memory.size)
> 9: Reached total allocation of 1535Mb: see help(memory.size)
> 10: Reached total allocation of 1535Mb: see help(memory.size)
> 11: In array(0, c(n, p)) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 12: In array(0, c(n, p)) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 13: In y[1L + 0L:(m - 1L) * (n + 1L)] <- x :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 14: In y[1L + 0L:(m - 1L) * (n + 1L)] <- x :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 15: In colnames(b) <- rownames(a) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 16: In colnames(b) <- rownames(a) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 17: In storage.mode(a) <- "double" :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 18: In storage.mode(a) <- "double" :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 19: In rownames(b) <- colnames(a) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 20: In rownames(b) <- colnames(a) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 21: In rownames(b) <- colnames(a) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 22: In rownames(b) <- colnames(a) :
>  Reached total allocation of 1535Mb: see help(memory.size)
>>
>
> method ="Matrix" (style ="B", the remaining part of code is identical as
> above)
>
> ?????????if (mean(as(last, "matrix")) < tol) { :
>  something lacking in TRUE/FALSE value
> ??????: ?????? 40 ????????? (??? warnings() ?????????)
> Warning messages:
> 1: In determinant(x, TRUE) : This version of the Matrix package returns
> |determinant(L)| instead of determinant(A), i.e., a
> *DIFFERENT* value.
> If still necessary, do change your code, following
> http://matrix.r-forge.r-project.org
>
> 2: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
>  NA/Inf is converted to the maximum positive value
> .........
> 23: In .local(object, ...) :
>  Reached total allocation of 1535Mb: see help(memory.size)
> 24: In .local(object, ...) :
>  Reached total allocation of 1535Mb: see help(memory.size)
>
> 25: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
>  NA/Inf is converted to the maximum positive value
> 40: the same
>
>
> method"spam"
> Loading required package: spam
> Package 'spam' is loaded. Spam version 0.22-0 (2010-06-08).
> Type demo( spam) for some demos, help( Spam) for an overview
> of this package.
> Help for individual functions is optained by adding the
> suffix '.spam' to the function name, e.g. 'help(chol.spam)'.
>
> Attaching package: 'spam'
>
>
>        The following object(s) are masked from package:Matrix :
>
>         norm
>
>
>        The following object(s) are masked from package:base :
>
>         backsolve,
>         forwardsolve
>
>> warnings()
> Warning messages:
> 1: package 'spam' was built under R version 2.10.1
> 2: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
>  NA/Inf is converted to the maximum positive value
> .....
> 16: the same
> 17: In t.spam(x) : Reached total allocation of 1535Mb: see help(memory.size)
> 18: In t.spam(x) : Reached total allocation of 1535Mb: see help(memory.size)
> 19: In optimize(sar.error.f, interval = interval, maximum = TRUE,  ... :
>  NA/Inf is converted to the maximum positive value
> ...
> 40:  the same
>
>
>
>
>
> On Fri, Jun 25, 2010 at 6:10 PM, Roger Bivand <Roger.Bivand at nhh.no> wrote:
>
>> On Fri, 25 Jun 2010, elaine kuo wrote:
>>
>>  Dear List,
>>>
>>> I am using errorsarlm (spdep) to measure SAR autocorrelation but failed as
>>> detailed below.
>>>
>>> system: windows XP and Vista
>>> RAM : 2G (XP) and 4G(Vista)
>>>
>>> sample size : 4500
>>> variable: 4
>>> nb10<-dnearneigh(coords,0,9)
>>>
>>> the error issue: out of memory of 180 Mb
>>>
>>> The similar messages in the archive were checked and tried but failed to
>>> solve the issues.
>>> 1. enlarging the R space to 2047 Mb
>>> 2. updating the neighboring unit from 0.5 to 9
>>>
>>> Please kindly help how to solve the problem, for I found SAR and CAR seem
>>> to
>>> be more reliable for autocorrelation than autocovariate.
>>>
>>
>>
>> No, you have not included the verbatim code leading to the memory problem,
>> nor the output of traceback() after the error, nor the output of
>> sessionInfo() to tell us which versions of packages you are using. This is
>> almost certainly nothing to do with dnearneigh(), unless the (0, 9)
>> thresholds include all n observations as neighbours for each neighbour, and
>> even then the neighbour object would not be very large.
>>
>> Did you read the help page for errorsarlm() or spautolm()? No? Then do so
>> and look carefully at the method= argument. You are almost certainly using
>> the default method, which uses dense matrices. Use your choice of sparse or
>> approximation alternatives, for example "Matrix" for updating sparse
>> Cholesky Jacobians, or "Chebyshev" for a fast approximate Jacobians.
>>
>> Using sparse matrix methods to overcome memory constraints has been
>> mentioned on this and other lists many times. With n=25000 ther is no
>> problem on a 1GB machine, so your problem doesn't even count as large.
>>
>> The list isn't really an alternative for reading help pages, and crucially
>> accessing the references given on the help pages. I'll add a link to
>> ?do_ldet on the function fitting pages, and add LeSage J and RK Pace (2009)
>> Introduction to Spatial Econometrics. CRC Press, Boca Raton too, as this is
>> the key reference that is required reading in this area.
>>
>> Hope this helps,
>>
>> Roger
>>
>>
>>
>>> Thanks
>>>
>>> Elaine
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at stat.math.ethz.ch
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>>
>> --
>> Roger Bivand
>> Economic Geography Section, Department of Economics, Norwegian School of
>> Economics and Business Administration, Helleveien 30, N-5045 Bergen,
>> Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
>> e-mail: Roger.Bivand at nhh.no
>>
>>
>
> 	[[alternative HTML version deleted]]
>
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no