[R-sig-Geo] Fitting SAR models in spdep (was: spatial 2SLS)
Roger Bivand
Roger.Bivand at nhh.no
Mon Dec 13 21:13:12 CET 2004
On Mon, 13 Dec 2004, Danlin Yu wrote:
> Dear Professor Roger Bivand:
>
> Yes, you are right, the total number of links is 686040, much more than
> 10*n (that is 3437) in the dataset.
>
> Some of the study area is really dense in data, but some is really
> sparse. So the chosen distance ensuring there won't be any non-zero links
> yield a large number of links. The actual reports from print.nb follows:
>
> Neighbour list object:
> Number of regions: 3437
> Number of nonzero links: 686040
> Percentage nonzero weights: 5.807515
> Average number of links: 199.6043
>
> Please keep me posted if the chol() problem is solved. Thanks. Danlin
Please try pre-release version 0.3-6 - source and Windows binary packages
on: http://spatial.nhh.no/R/spdep/
It would be good if this works. I think, though, that a graph-based
neighbour list (SOI graph, for example) would be a more natural choice
than distances, but for testing, this neighbour list is very suitable.
Roger
>
> On Mon, 13 Dec 2004, Roger Bivand wrote:
>
> > On Mon, 13 Dec 2004, Danlin Yu wrote:
> >
> > > Dear Professor Roger Bivand:
> > >
> > > Yes, I was talking about the two SAR fitting functions. I just did a
> > > very quick run on a dataset with 3437 observations (those are point data,
> > > and I generate the listw object using dnearneigh() - not sure whether the
> > > result is symmetric or asymmetric).
> >
> > How many links are there in the neighbour list? Distance neighbours are
> > symmetric, but if points are densely placed, there may be parts of your
> > study area with many neighbours. Use print.nb() to find out how many.
> >
> > > However, I tried both "sparse" and
> > > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > > given as follow:
> > >
> > > For "sparse" method:
> > > "Error in spwdet(sparseweights, rho = rho, debug = debug) :
> > > Suspicious allocation of memory in sparse functions
> > > ...bailing out! Save workspace and quit R!"
> >
> > Yes, this is the memory bug. It does happen on most/all GCC compiled
> > versions, but I've seen BSD versions that work - it depends on the memory
> > allocation code, but it has not been possible to solve. I am planning to
> > drop "sparse" before long, unless the memory bug is resolved.
> >
> > >
> > > For "SparseM" method:
> > > "Error in .local(x, ...) : Increase tmpmax
> >
> > This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
> > as only 10 times the number of rows. I think you have more neighbour links
> > than 10*n, and that means I need to find a way of passing a better value
> > through. Please let me know, this looks do-able.
> >
> > Roger
> >
> > > Error in det(chol(x)) : Unable to find the argument "x" in selecting
> > > a method for function "det" "
> > >
> > > By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > > I got warnings. I will keep trying and keep you posted as I progress.
> > > Thanks.
> > >
> > >
> > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > >
> > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > >
> > > > > Dear Mihai Nica:
> > > > >
> > > > > I think it is a memory issue, too. I ran into the same problem when
> > > > > using around 3000 observations. Although I used memory.size() to increase
> > > > > the memroy allocation size, spdep took too long to run.
> > > > >
> > > >
> > > > I think you mean the two SAR fitting functions, lagsarlm() and
> > > > errorsarlm()? The default method is "eigen", which does need plenty of
> > > > memory with large n (because it both solves the eigenproblem for an nxn
> > > > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > > > method using code from Netlib called sparse from UCB, which works on some
> > > > platforms, but not others, and has an unresolved memory bug.
> > > >
> > > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > > dropping each of the X variables in turn, as well as the specified model,
> > > > so that it fits k models, not just 1. The function calls for the lag
> > > > models are cheaper than the error models, though.
> > > >
> > > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > > graphe-based and distance bands), and even symmetric inverse distances,
> > > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > > to symmetric, and the others are symmetric), but absolutely not for
> > > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > > tables).
> > > >
> > > > I would be very interested in hearing whether this approach is helpful,
> > > > and how this scales.
> > > >
> > > > Roger
> > > >
> > > > > The problem, I think, might be due to the fact that spdep's spatial
> > > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > > likelihood estimation of very large spatial autoregressive models: a
> > > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > > will be great.
> > > > >
> > > > > Danlin
> > > > >
> > > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > > >
> > > > > > Greetings:
> > > > > >
> > > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > > check my results would be great.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mihai Nica
> > > > > > Jackson State University
> > > > > > 155 B Parkhurst Dr.
> > > > > > Jackson, MS 39202
> > > > > > 601 914 0361
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > > >
> > > > > >
> > > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > > and
> > > > > > > not up to official R standards yet, but they definitely work.
> > > > > > > The functions include:
> > > > > > > - standard 2SLS (or IV estimation)
> > > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > > me directly.
> > > > > > > L.
> > > > > > >
> > > > > > >
> > > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > > >
> > > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep? If not,
> > > > > > > > is it
> > > > > > > > available anywhere else at this time? (besides in SpaceStat).
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Darla Munroe
> > > > > > > >
> > > > > > > >
> > > > > > > > [[alternative HTML version deleted]]
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > R-sig-Geo mailing list
> > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > R-sig-Geo mailing list
> > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > R-sig-Geo mailing list
> > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > >
> > > > >
> > > > > Sincerely,
> > > > > Danlin Yu
> > > > >
> > > > > ----------------------------------
> > > > > Lecturer, Ph.D. Candidate
> > > > > Department of Geography
> > > > > University of Wisconsin, Milwaukee
> > > > > Tel: (414)229-3943
> > > > > Fax: (414)229-3981
> > > > > Email: danlinyu at uwm.edu
> > > > >
> > > > > _______________________________________________
> > > > > R-sig-Geo mailing list
> > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > >
> > > >
> > > > --
> > > > Roger Bivand
> > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > e-mail: Roger.Bivand at nhh.no
> > > >
> > > >
> > >
> > > Sincerely,
> > > Danlin Yu
> > >
> > > ----------------------------------
> > > Lecturer, Ph.D. Candidate
> > > Department of Geography
> > > University of Wisconsin, Milwaukee
> > > Tel: (414)229-3943
> > > Fax: (414)229-3981
> > > Email: danlinyu at uwm.edu
> > > ----------------------------------
> > >
> > >
> >
> > --
> > Roger Bivand
> > Economic Geography Section, Department of Economics, Norwegian School of
> > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > e-mail: Roger.Bivand at nhh.no
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at stat.math.ethz.ch
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
> Sincerely,
> Danlin Yu
>
> ----------------------------------
> Lecturer, Ph.D. Candidate
> Department of Geography
> University of Wisconsin, Milwaukee
> Tel: (414)229-3943
> Fax: (414)229-3981
> Email: danlinyu at uwm.edu
> ----------------------------------
>
>
--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand at nhh.no
More information about the R-sig-Geo
mailing list