[R-sig-Geo] error in errorsarlm

[Roger Bivand]

On Sat, 17 Jul 2010, elaine kuo wrote:

> Dear,
>
> Thank you for the help.
>
> 1. distance threshold
> It is 9. (Roger, you are right and sorry to confuse you)

But 9 what, metres, degrees? Should your code be: 
cbind(datam$lat,datam$lon), as the normal ordering is eastings then 
northings, not the opposite?

>
> 9 is the cutting point for avoiding empty neighbour sets.
> (If the distance threshold is 8, there will be empty ones.)

This is not a good reason, as graph neighbours can be sparse and link over 
longer distances.

>
> 2. dataset
> length(nb10)=4873.
> sum(card(nb10)=885478)
>

This gives an average of 181.7 neighbours per observation, and is the root 
of the problem. The Chebyshev method uses a set of low powers of the 
weights, which fill in quickly for large average numbers of neighbours. 
Did you try the Matrix method (your error message suggests that you may 
have done, but the code does not)?

> 3. memory size
> the memory limit is 1535 Mb, which is much larger than 121.8 Mb.
>

Windows (your OS?) memory handling is very poor for larger objects, and 
the 121Mb is extra memory needed. However, the main problem is your design 
of weights, which are far too dense to make sense. Choose an alternative 
weights definition and try again.

Roger

>
> Elaine
>
>
> Code
>
>> library(ncf)
>> library(spdep)
>>
>> # Define coordinates, neighbours, and spatial weights
>> coords<-cbind(datam$lat,datam$lon)
>> coords<-as.matrix(coords)
>>
>> #Define neighbourhood (here distance 1.5)
>> nb10<-dnearneigh(coords,0,9)
>> nb10.w<-nb2listw(nb10, glist=NULL, style="W", zero.policy=FALSE)
>> sem.nb10.w <-
>>
> errorsarlm(datam$WinterM_ratio~datam$topo_mean+datam$coast+datam$prec_max
> +datam$temp_min+datam$evi_min, listw=nb10.w,
> method="Chebyshev",tol.solve=1.0e-17)
> error: cannot allocate vector of size 121.8 Mb
>
> sem.nb10.w <-
>>
> errorsarlm(datam$WinterM_ratio~datam$topo_mean+datam$coast+datam$prec_max
> +datam$temp_min+datam$evi_min, listw=nb10.w,
> method="Chebyshev",zero.policy=TRUE,tol.solve=1.0e-17)
> error in switch(Generic, `+` = , `-` = { :
> Cholmod error 'out of memory' at file:../Core/cholmod_memory.c, line 148
>
>
> On Sat, Jul 17, 2010 at 3:52 AM, Roger Bivand <Roger.Bivand at nhh.no> wrote:
>
>> On Fri, 16 Jul 2010, elaine kuo wrote:
>>
>>  Dear List,
>>>
>>> I encountered an error message when trying to use Chebyshev as method for
>>> SAR.
>>> Please kindly help and thanks.
>>>
>>
>> Beyond what Danlin said, I am very unsure where to start. Please provide
>> adequate information about your input data, specifically the number of
>> observations, and the number of links in the neighbour list
>>
>> length(nb10)
>> sum(card(nb10))
>>
>> In addition, the implicit discription of nb10 looks confusing (is the
>> distance threshold 1.5, 9, or 10?). The ordering of lat and lon looks wrong.
>> You are not using Great Circle distances (and probably should?). So memory
>> issues in errorsarlm are the least of your problems. My guess is that with
>> dnearneigh() and a threshold of 9 degrees, you are including most of a large
>> data set in the neighbour list, and if n is large, this will give a very
>> large almost dense set of spatial weights, which was not what you intended.
>>
>> Get the weights correct first, and the rest will likely follow.
>>
>> Roger
>>
>>
>>
>>> package spdep was updated on 07162010
>>> system: windows vista home 32-bit
>>> R version 2.10.0 (20091026)
>>>
>>> Elaine
>>>
>>> Code
>>>
>>>> library(ncf)
>>>> library(spdep)
>>>>
>>>> # Define coordinates, neighbours, and spatial weights
>>>> coords<-cbind(datam$lat,datam$lon)
>>>> coords<-as.matrix(coords)
>>>>
>>>> #Define neighbourhood (here distance 1.5)
>>>> nb10<-dnearneigh(coords,0,9)
>>>> nb10.w<-nb2listw(nb10, glist=NULL, style="W", zero.policy=FALSE)
>>>> sem.nb10.w <-
>>>>
>>> errorsarlm(datam$WinterM_ratio~datam$topo_mean+datam$coast+datam$prec_max
>>> +datam$temp_min+datam$evi_min, listw=nb10.w,
>>> method="Chebyshev",tol.solve=1.0e-17)
>>> error: cannot allocate vector of size 121.8 Mb
>>>
>>>  sem.nb10.w <-
>>>>
>>> errorsarlm(datam$WinterM_ratio~datam$topo_mean+datam$coast+datam$prec_max
>>> +datam$temp_min+datam$evi_min, listw=nb10.w,
>>> method="Chebyshev",zero.policy=TRUE,tol.solve=1.0e-17)
>>> error in switch(Generic, `+` = , `-` = { :
>>>  Cholmod error 'out of memory' at file:../Core/cholmod_memory.c, line 148
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at stat.math.ethz.ch
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>>
>> --
>> Roger Bivand
>> Economic Geography Section, Department of Economics, Norwegian School of
>> Economics and Business Administration, Helleveien 30, N-5045 Bergen,
>> Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
>> e-mail: Roger.Bivand at nhh.no
>>
>>
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no