[R-sig-Geo] inverse distance weights matrix
Roger Bivand
Roger.Bivand at nhh.no
Tue Apr 11 17:21:29 CEST 2006
On Tue, 11 Apr 2006, Griffin, Terry W wrote:
> Thanks Roger,
>
> I've been working on creating inverse distance matrices; and have been
> successful in this first step. However, as you suggested, the
> connectedness of these matrices will cause procedures to be slow, even
> prohibitive with my computer (P4 3Gz 1GB RAM) (ran out of RAM).
>
With an almost dense weights matrix and large n, this is difficult to
avoid.
> I've tried a simpler first order queen weights matrix to see if I could
> get the errorsarlm model to run; but come to a different problem and an
> error message:
>
> source("spat.cot.R")
> Error in solve.default(asyvar, tol = tol.solve) :
> system is computationally singular: reciprocal condition number
> = 1.16119e-019
> > traceback()
> 6: solve.default(asyvar, tol = tol.solve)
> 5: solve(asyvar, tol = tol.solve)
> 4: errorsarlm(YLD02 ~ CLAY + DIST + PEG + ROT + SUN, data = cotton,
> q1)
> 3: eval.with.vis(expr, envir, enclos)
> 2: eval.with.vis(ei, envir)
> 1: source("spat.cot.R")
>
> where "spat.cot.R" contains:
> spatcot.err<-errorsarlm(YLD02~CLAY+DIST+PEG+ROT+SUN,data=cotton, q1)
>
> Do I need to set my tol to a different value or am I having other
> issues?
Yes, as on the help page, set tol.solve=1e-19, or (at your choice) scale
the right hand side variables to bring the coefficients nearer to a -1 to
+1 range (like lambda), usually by multiplying by some power of 10.
Roger
>
> Thank you,
>
> Terry
>
>
>
>
> -----Original Message-----
> From: Roger Bivand [mailto:Roger.Bivand at nhh.no]
> Sent: Saturday, March 18, 2006 3:07 PM
> To: Griffin, Terry W
> Cc: r-sig-geo at stat.math.ethz.ch
> Subject: RE: [R-sig-Geo] inverse distance weights matrix
>
> On Sat, 18 Mar 2006, Griffin, Terry W wrote:
>
> > Thank you Roger,
> >
> > I've posted the original shape file for the dataset on the website at:
> > http://web.ics.purdue.edu/~twgriffi/R_matrix.htm
> >
> > One shape file is projected while the other is not; however both sets
> of
> > *.dbf are identical and contain both decimal degrees and Northing and
> > Easting in meters.
> >
> > I chose a <=75 meter distance threshold and a power=1.
> >
> > The final inverse distance matrix was row standardized so it is not
> > symmetric. The weight characteristics you list below are nearly
> > identical to those that SpaceStat provides.
>
> OK - it looks like this at the moment:
>
> library(rgdal)
> cotton_proj <- readOGR(".", "cotton_proj") # for shapefiles in the
> current
> # working directory, first argument
> plot(cotton_proj, cex=0.2, axes=TRUE)
> library(spdep)
> d75 <- dnearneigh(coordinates(cotton_proj), 0, 75)
> dlist <- nbdists(d75, coordinates(cotton_proj))
> idlist <- lapply(dlist, function(x) 1/x)
> w75 <- nb2listw(d75, glist=idlist, style="W")
>
> Characteristics of d75:
>
> > d75
> Neighbour list object:
> Number of regions: 2451
> Number of nonzero links: 1182994
> Percentage nonzero weights: 19.69228
> Average number of links: 482.6577
>
> But:
>
> > inMAT <- matrix(scan("cottonID75wm", sep=","), nrow=2451, ncol=2451,
> + byrow=TRUE)
> > range(rowSums(inMAT))
> [1] 0.8159028 1.9115231
> > d75MAT <- nb2mat(d75, glist=idlist, style="W")
> > range(rowSums(d75MAT))
> [1] 1 1
> > w75 <- nb2listw(d75, glist=idlist, style="W")
> > range(sapply(w75$weights, sum))
> [1] 1 1
>
> So something in the construction of your matrix wasn't securing row
> standardisation. The values do differ, this is the upper left corner:
>
> > d75MAT[1:5,1:5]
> [,1] [,2] [,3] [,4] [,5]
> 1 0.00000000 0.06735710 0.03227068 0.02182278 0.01660474
> 2 0.05975093 0.00000000 0.05493976 0.02863058 0.01954588
> 3 0.02615210 0.05019078 0.00000000 0.05461937 0.02771729
> 4 0.01639324 0.02424508 0.05062943 0.00000000 0.05216365
> 5 0.01172946 0.01556467 0.02416008 0.04905229 0.00000000
> > inMAT[1:5,1:5]
> [,1] [,2] [,3] [,4] [,5]
> [1,] 0.000000 0.131420 0.061844 0.041726 0.031848
> [2,] 0.101440 0.000000 0.090156 0.047180 0.032441
> [3,] 0.040305 0.076122 0.000000 0.083545 0.042787
> [4,] 0.023708 0.034730 0.072836 0.000000 0.076431
> [5,] 0.016296 0.021506 0.033593 0.068830 0.000000
> > rowSums(inMAT[1:5,])
> [1] 1.911523 1.658689 1.533153 1.440908 1.379051
> > rowSums(d75MAT[1:5,])
> 1 2 3 4 5
> 1 1 1 1 1
>
> They are not the same matrices, although the differences are very small
> -
> here I've used the shapefile points, which are stored as binary floating
>
> point, rather than using the DBF values, which are a possibly rounded
> character representation, and that might be enough to cause the
> difference. I think you should be able to generate w75, and that it
> should
> do what you need, but I'd be grateful for feedback on how you get on.
>
> Best wishes,
>
> Roger
>
> PS: with numbers of neighbours between 177 and 678 (both in your matrix
> and in d75), even allowing for the dampening effect of inverse distance,
>
> most analyses will be slow. 75m means that many sites are related to
> many
> in your case. Doing:
>
> symbols(408725, 3660800, circles=75, inches=FALSE, fg="red", add=TRUE)
>
> on top of the earlier plot shows your chosen neighbourhood for an
> arbitrary point.
>
> >
> > I really appreciate this. Thank you,
> >
> > Terry
> >
> >
> >
> > -----Original Message-----
> > From: Roger Bivand [mailto:Roger.Bivand at nhh.no]
> > Sent: Friday, March 17, 2006 5:50 PM
> > To: Griffin, Terry W
> > Cc: r-sig-geo at stat.math.ethz.ch
> > Subject: RE: [R-sig-Geo] inverse distance weights matrix
> >
> > Terry:
> >
> > http://web.ics.purdue.edu/~twgriffi/mat4rf
> >
> > and
> >
> > http://web.ics.purdue.edu/~twgriffi/mat4rs
> >
> > throw access errors.
> >
> > Do you have the coordinates and your specifications for the inverse
> > weighted neighbours (distance threshold, power if any)?
> >
> > inMAT <- matrix(scan("cottonID75wm", sep=","), nrow=2451, ncol=2451,
> > byrow=TRUE)
> >
> > does read the matrix - the weights aren't symmetric, are they?
> >
> > From the weights matrix I get:
> >
> > inMATlw <- mat2listw(inMAT)
> >
> > print(is.symmetric.nb(inMATlw$neighbours))
> > [1] TRUE
> >
> > print(inMATlw$neighbours)
> > Neighbour list object:
> > Number of regions: 2451
> > Number of nonzero links: 1182982
> > Percentage nonzero weights: 19.69208
> > Average number of links: 482.6528
> >
> > If you could post the coordinates and your specifications, I could see
> > how
> > to reproduce the weights within R if you like.
> >
> > Roger
> >
> >
> > On Fri, 17 Mar 2006, Griffin, Terry W wrote:
> >
> > >
> > > The MATLAB routines are the Spatial Econometrics Toolbox format. I
> > > tried the read.dat2listw() in R; the code and error message for a
> > small
> > > matrix example is below; this uses a full matrix from MATLAB
> > (realizing
> > > that "read.dat2listw" is intended for sparse matrices).
> > >
> > > > mattry<-read.dat2listw("mat4rf")
> > > Error in "[.default"(sn, , 1) : incorrect number of dimensions
> > > > traceback()
> > > 5: NextMethod("[")
> > > 4: "[.factor"(sn, , 1)
> > > 3: sn[, 1]
> > > 2: unique(sn[, 1])
> > > 1: read.dat2listw("mat4rf")
> > >
> > >
> > > I'm not able to use dlmwrite in MATLAB to export the sparse matrix
> so
> > I
> > > tried to import the full matrix into R (code is above). The MATLAB
> > code
> > > I'm using came from this listserve archive. The MATLAB code and
> error
> > > message are below. I'm able to use dlmwrite to create the MATLAB
> full
> > > matrix (used above in the R code).
> > >
> > > load testmat.GWT; A = (spconvert(testmat)); %have to change name in
> > two
> > > locations on this line
> > > n=size(A); nobs=n(1);
> > > % Normalize matrix (adapted form LeSage 1999)
> > > wt = sparse(A);
> > > [i1,j1,s1]=find(wt);
> > > rsum=sum(wt);
> > > for i=1:nobs;
> > > ind=find(i1==i);
> > > s1(ind,1)=s1(ind,1)/rsum(1,i);
> > > end;
> > > [m,n] = size(wt);
> > > W = sparse(i1,j1,s1,m,n);
> > > S = sparse(W);
> > > dlmwrite('mat4rst', S); %give the new sparse weight matrix a
> permanent
> > > name for R
> > > ??? Error using ==> sprintf
> > > Function is not defined for sparse inputs.
> > >
> > > Error in ==> dlmwrite at 172
> > > str = sprintf(format,m(i,:));
> > >
> > > If anyone wants to give it a try, I've put the small example
> matrices
> > > and the "real" matrix up on my website at:
> > > http://web.ics.purdue.edu/~twgriffi/R_matrix.htm
> > >
> > > My objective is to use R for my spatial data analysis using inverse
> > > distance weights matrices. I'm struggling with the inverse distance
> > > matrices. I hope to either create these inverse distance matrices
> in
> > > SpaceStat then import into R or create the matrices in R.
> > >
> > > Thank you,
> > >
> > > Terry
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Roger Bivand [mailto:Roger.Bivand at nhh.no]
> > > Sent: Friday, March 17, 2006 2:43 PM
> > > To: Griffin, Terry W
> > > Cc: r-sig-geo at stat.math.ethz.ch
> > > Subject: Re: [R-sig-Geo] inverse distance weights matrix
> > >
> > > On Thu, 16 Mar 2006, Griffin, Terry W wrote:
> > >
> > > > Greetings,
> > > >
> > > >
> > > >
> > > > I'm migrating from MATLAB to R and am having trouble converting
> > > inverse
> > > > distance weights matrix into a form usable by R. I've created the
> > > > matrix in SpaceStat (in *.FMT format) and converted to *.GWT
> format
> > > and
> > > > also have it in MATLAB *.m format.
> > > >
> > >
> > > I wonder whether your Matlab data are in the Spatial Econometrics
> > > Toolbox
> > > format - if they are, you could try the read.dat2listw() function
> > > mentioned on the read.gwt2nb function help page.
> > > >
> > > >
> > > > I've tried using the read.gwt2nb in R, but without success. There
> > are
> > > > nearly 2,451 observations with an average of 482 linkages. About
> 20
> > %
> > > > have nonzero weights. Any suggestions are appreciated.
> > >
> > > Without details of what you mean by "without success", it's hard to
> > > offer
> > > any advice. Could you create a small example of the problems you are
>
> > > seeing, and repost with copies of the commands you give and any
> error
> > > messages?
> > >
> > > If you are seeing errors ("Error in ..." messages from R), please
> also
> > > type traceback() at the prompt and report the output. You could also
> > put
> > > the uncooperative GWT file on a website for others to try to see
> what
> > is
> > > going on.
> > >
> > > >
> > > >
> > > >
> > > > Thank you,
> > > >
> > > > Terry
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Terry W. Griffin
> > > >
> > > > Graduate Research Assistant
> > > >
> > > > Agricultural Economics
> > > >
> > > > Purdue University
> > > >
> > > > 403 W State St
> > > >
> > > > West Lafayette, IN 47907
> > > >
> > > > 765-494-4257
> > > >
> > > > http://web.ics.purdue.edu/~twgriffi/
> > > > <http://web.ics.purdue.edu/~twgriffi/>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > [[alternative HTML version deleted]]
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo at stat.math.ethz.ch
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > >
> > >
> > >
> >
> >
>
>
--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no
More information about the R-sig-Geo
mailing list