[R-sig-Geo] Fitting SAR models in spdep (was: spatial 2SLS)

Danlin Yu danlinyu at csd.uwm.edu
Mon Dec 13 21:52:41 CET 2004


Dear Professor Roger Bivand:

    Actually, I just tried lagsarlm() using the same dataset and same
weight list, but it did not go through for two reasons: first, the memory
is still a problem; second, it seems that det() function did not work well
on the lagsarlm(). The actually error message follows:

 Error: cannot allocate vector of size 5386 Kb
 In addition: Warning message: 
 Reached total allocation of 446Mb: see help(memory.size) 
 Error in det(chol((I - rho * W), tmpmax = tmpmax)) : 
    Unable to find the argument "x" in selecting a method for function "det"


On Mon, 13 Dec 2004, Roger Bivand wrote: 

> On Mon, 13 Dec 2004, Danlin Yu wrote:
> 
> > Dear Professor Roger Bivand:
> > 
> >     Yes, it worked! The same set of data (3437), and same neighbor list
> > (listw), on my computer (a laptop), for the errorsarlm(), using method
> > "SparseM", it worked within 5 minutes, no error reported. Is the
> > lagsarlm() the same?
> 
> Yes, I hope so, the same value is passed to tmpmax, the sum of the number 
> of neighbour links (the off-diagonal non-zero values) and n (the 
> diagonal). Five minutes is not fast, but it's not bad, and the results 
> should be reliable. The big advantage of using SparseM is that Professors 
> Roger Koenker and Pin Ng are maintaining the package actively, and using 
> it for plenty of other work, so any problems are very likely to be found 
> quickly.
> 
> If nobody reports serious problems, I'll submit 0.3-6 to CRAN on Tuesday.
> 
> Roger
> 
> > 
> > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > 
> > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > 
> > > > Dear Professor Roger Bivand:
> > > > 
> > > >    Yes, you are right, the total number of links is 686040, much more than
> > > > 10*n (that is 3437) in the dataset.
> > > > 
> > > >    Some of the study area is really dense in data, but some is really
> > > > sparse. So the chosen distance ensuring there won't be any non-zero links
> > > > yield a large number of links. The actual reports from print.nb follows:
> > > > 
> > > > Neighbour list object:
> > > > Number of regions: 3437 
> > > > Number of nonzero links: 686040 
> > > > Percentage nonzero weights: 5.807515 
> > > > Average number of links: 199.6043
> > > > 
> > > >    Please keep me posted if the chol() problem is solved. Thanks. Danlin
> > > 
> > > Please try pre-release version 0.3-6 - source and Windows binary packages 
> > > on: http://spatial.nhh.no/R/spdep/
> > > 
> > > It would be good if this works. I think, though, that a graph-based 
> > > neighbour list (SOI graph, for example) would be a more natural choice 
> > > than distances, but for testing, this neighbour list is very suitable.
> > > 
> > > Roger
> > > 
> > > > 
> > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > > 
> > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > > 
> > > > > > Dear Professor Roger Bivand:
> > > > > > 
> > > > > >     Yes, I was talking about the two SAR fitting functions. I just did a
> > > > > > very quick run on a dataset with 3437 observations (those are point data,
> > > > > > and I generate the listw object using dnearneigh() - not sure whether the
> > > > > > result is symmetric or asymmetric). 
> > > > > 
> > > > > How many links are there in the neighbour list? Distance neighbours are 
> > > > > symmetric, but if points are densely placed, there may be parts of your 
> > > > > study area with many neighbours. Use print.nb() to find out how many.
> > > > > 
> > > > > > However, I tried both "sparse" and
> > > > > > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > > > > > given as follow:
> > > > > > 
> > > > > >     For "sparse" method:
> > > > > >     "Error in spwdet(sparseweights, rho = rho, debug = debug) : 
> > > > > >         Suspicious allocation of memory in sparse functions
> > > > > >          ...bailing out! Save workspace and quit R!"
> > > > > 
> > > > > Yes, this is the memory bug. It does happen on most/all GCC compiled 
> > > > > versions, but I've seen BSD versions that work - it depends on the memory 
> > > > > allocation code, but it has not been possible to solve. I am planning to 
> > > > > drop "sparse" before long, unless the memory bug is resolved.
> > > > > 
> > > > > > 
> > > > > >     For "SparseM" method:
> > > > > >     "Error in .local(x, ...) : Increase tmpmax
> > > > > 
> > > > > This is the problem: tmpmax is defined in chol() for "matrix.csr" objects 
> > > > > as only 10 times the number of rows. I think you have more neighbour links 
> > > > > than 10*n, and that means I need to find a way of passing a better value 
> > > > > through. Please let me know, this looks do-able.
> > > > > 
> > > > > Roger
> > > > > 
> > > > > >      Error in det(chol(x)) : Unable to find the argument "x" in selecting
> > > > > >      a method for function "det" "
> > > > > > 
> > > > > >     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > > > > > I got warnings. I will keep trying and keep you posted as I progress.
> > > > > > Thanks.
> > > > > > 
> > > > > > 
> > > > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > > > > 
> > > > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > > > > 
> > > > > > > > Dear Mihai Nica:
> > > > > > > > 
> > > > > > > >     I think it is a memory issue, too. I ran into the same problem when
> > > > > > > > using around 3000 observations. Although I used memory.size() to increase
> > > > > > > > the memroy allocation size, spdep took too long to run.
> > > > > > > > 
> > > > > > > 
> > > > > > > I think you mean the two SAR fitting functions, lagsarlm() and 
> > > > > > > errorsarlm()? The default method is "eigen", which does need plenty of 
> > > > > > > memory with large n (because it both solves the eigenproblem for an nxn 
> > > > > > > matrix, and later inverts an nxn matrix). There has always been a "sparse" 
> > > > > > > method using code from Netlib called sparse from UCB, which works on some 
> > > > > > > platforms, but not others, and has an unresolved memory bug. 
> > > > > > > 
> > > > > > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > > > > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > > > > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > > > > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > > > > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > > > > > dropping each of the X variables in turn, as well as the specified model,
> > > > > > > so that it fits k models, not just 1. The function calls for the lag
> > > > > > > models are cheaper than the error models, though.
> > > > > > > 
> > > > > > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > > > > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > > > > > graphe-based and distance bands), and even symmetric inverse distances,
> > > > > > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > > > > > to symmetric, and the others are symmetric), but absolutely not for
> > > > > > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > > > > > tables).
> > > > > > > 
> > > > > > > I would be very interested in hearing whether this approach is helpful, 
> > > > > > > and how this scales.
> > > > > > > 
> > > > > > > Roger
> > > > > > > 
> > > > > > > >     The problem, I think, might be due to the fact that spdep's spatial
> > > > > > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > > > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > > > > > likelihood estimation of very large spatial autoregressive models: a
> > > > > > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > > > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > > > > > will be great.
> > > > > > > > 
> > > > > > > >     Danlin
> > > > > > > > 
> > > > > > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > > > > > > 
> > > > > > > > > Greetings:
> > > > > > > > > 
> > > > > > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > > > > > check my results would be great.
> > > > > > > > > 
> > > > > > > > > Thanks,
> > > > > > > > > 
> > > > > > > > > Mihai Nica
> > > > > > > > > Jackson State University
> > > > > > > > > 155 B Parkhurst Dr.
> > > > > > > > > Jackson, MS 39202
> > > > > > > > > 601 914 0361
> > > > > > > > > 
> > > > > > > > > ----- Original Message ----- 
> > > > > > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > > > > > and
> > > > > > > > > > not up to official R standards yet, but they definitely work.
> > > > > > > > > > The functions include:
> > > > > > > > > > - standard 2SLS (or IV estimation)
> > > > > > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > > > > > me directly.
> > > > > > > > > > L.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > > > > > >
> > > > > > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > > > > > > is it
> > > > > > > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Darla Munroe
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [[alternative HTML version deleted]]
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > >
> > > > > > > > > 
> > > > > > > > > _______________________________________________
> > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > 
> > > > > > > > 
> > > > > > > >     Sincerely,
> > > > > > > >     Danlin Yu
> > > > > > > > 
> > > > > > > > ----------------------------------
> > > > > > > > Lecturer, Ph.D. Candidate
> > > > > > > > Department of Geography
> > > > > > > > University of Wisconsin, Milwaukee
> > > > > > > > Tel: (414)229-3943
> > > > > > > > Fax: (414)229-3981
> > > > > > > > Email: danlinyu at uwm.edu
> > > > > > > > 
> > > > > > > > _______________________________________________
> > > > > > > > R-sig-Geo mailing list
> > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > 
> > > > > > > 
> > > > > > > -- 
> > > > > > > Roger Bivand
> > > > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > > > e-mail: Roger.Bivand at nhh.no
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > >     Sincerely,
> > > > > >     Danlin Yu
> > > > > > 
> > > > > > ----------------------------------
> > > > > > Lecturer, Ph.D. Candidate
> > > > > > Department of Geography
> > > > > > University of Wisconsin, Milwaukee
> > > > > > Tel: (414)229-3943
> > > > > > Fax: (414)229-3981
> > > > > > Email: danlinyu at uwm.edu
> > > > > > ----------------------------------
> > > > > > 
> > > > > > 
> > > > > 
> > > > > -- 
> > > > > Roger Bivand
> > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > e-mail: Roger.Bivand at nhh.no
> > > > > 
> > > > > _______________________________________________
> > > > > R-sig-Geo mailing list
> > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > 
> > > > 
> > > >     Sincerely,
> > > >     Danlin Yu
> > > > 
> > > > ----------------------------------
> > > > Lecturer, Ph.D. Candidate
> > > > Department of Geography
> > > > University of Wisconsin, Milwaukee
> > > > Tel: (414)229-3943
> > > > Fax: (414)229-3981
> > > > Email: danlinyu at uwm.edu
> > > > ----------------------------------
> > > > 
> > > > 
> > > 
> > > -- 
> > > Roger Bivand
> > > Economic Geography Section, Department of Economics, Norwegian School of
> > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > e-mail: Roger.Bivand at nhh.no
> > > 
> > > 
> > 
> >     Sincerely,
> >     Danlin Yu
> > 
> > ----------------------------------
> > Lecturer, Ph.D. Candidate
> > Department of Geography
> > University of Wisconsin, Milwaukee
> > Tel: (414)229-3943
> > Fax: (414)229-3981
> > Email: danlinyu at uwm.edu
> > ----------------------------------
> > 
> > 
> 
> -- 
> Roger Bivand
> Economic Geography Section, Department of Economics, Norwegian School of
> Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> e-mail: Roger.Bivand at nhh.no
> 
> 

    Sincerely,
    Danlin Yu

----------------------------------
Lecturer, Ph.D. Candidate
Department of Geography
University of Wisconsin, Milwaukee
Tel: (414)229-3943
Fax: (414)229-3981
Email: danlinyu at uwm.edu




More information about the R-sig-Geo mailing list