[R-sig-Geo] How to efficiently generate data of neighboring points

Roger Bivand Roger@B|v@nd @end|ng |rom nhh@no
Fri Jun 5 10:51:20 CEST 2020 

On Fri, 5 Jun 2020, Lom Navanyo wrote:

> I fully agree with you and appreciate the listed benefits of not taking
> things private. I was just trying to be sure the forum here is appropriate
> and receptive of a beginner like me.
>
> To be more explicit with regards to my observations, y is amount of 
> water withdrawal from wells and an important variable in x is (height 
> of) water level in the wells. These are end of year figures. I am using 
> the aggregations (sum for y and mean for water level) by band as spatial 
> neighborhood variables. There will be one or two indicator variables 
> also in x. I hope these do not present additional hurdles.

There are several further questions. If water level is measured at 
end-of-year, it is instantaneous at that point, and will depend on level a 
year earlier plus inflow from the movements of the water table 
(precipitation, soils and surface geology, maybe geology if deeper wells), 
minus evaporation (if an open well) and extraction. However, your y 
(extraction) is probably measured over an interval (1 Jan - 31 Dec?). It 
does not depend on level unless level is 0, but depends on the closeness 
of people extracting water for domestic, agricultural or other use.

All else equal, you would expect changes in the level in a well to depend 
on inputs, evaporation and extraction, and extraction at that well and 
other nearby wells (which may experience falls in the ground water table 
level not because the water was extracted from those wells, but at 
neighbouring wells. You may also see users shifting to neerby wells if 
their closest well runs dry.

So you probably need to start with a deterministic hydrological model, and 
you need much more information about who extracts and why. Say in India, 
you would also need price data - apparently free water has led to 
over-extraction.

So I would advise against any spatial econometric analysis of the data you 
have, because so much is going on in the system as a whole that you cannot 
control if all the data you have is as you describe. I also understand 
better why well water level is endogeneous, but am sure that IV will not 
help, since the level is being driven partly by a deterministic 
hydrological system which differs from well to well, and extraction varies 
by demand.

Has anyone worked with this kind of data? Any ideas or contributions more 
helpful than the above?

Roger

>
> I am thinking Proximity is relevant in testing spatial
> dependency/externality.
>
> I will consider splm package  and the SLX model.
>
> Thank you.
> ---------------
> Lom
>
> On Thu, Jun 4, 2020 at 2:52 PM Roger Bivand <Roger.Bivand using nhh.no> wrote:
>
>> On Thu, 4 Jun 2020, Lom Navanyo wrote:
>>
>>> Thank you. Yes, the OLS is biased and my plan is to use a 2SLS approach.
>> I
>>> have a variable I intend to use as an IV for y.
>>> I have seen a few papers use this approach. Will this approach not
>> correct
>>> for the endogeneity?
>>>
>>> Actually, I am not sure if this is a right forum or perhaps if it's
>>> appropriate or acceptable to you to take this one-on-one with you for
>> help:
>>
>> I do not offer private help. That would presuppose that one person has the
>> answer. It would also presuppose that all exchanges are only read by the
>> original poster and direct participants, while in fact others may join in,
>> or follow a thread, or find the thread by searching: google supports the
>> list:r-sig-geo search tag. If the thread goes private, that search is
>> fruitless.
>>
>>> My model actually looks like this: y= f(y, x)  + e.
>>> Aside the endogeneity of y (which I intend to instrument by another
>>> variable z), there is simultaneity between y and x.
>>> I intend to use the lag of x as instrument for x.  Given that I am
>> seeking
>>> to test spatial dependency, do you see some fatal flaws with my approach?
>>>
>>
>> What is the support of your observations, point, or are they aggregations?
>> Why may proximity make a difference - often, apparent spatial
>> autocorrelation is caused by observing inappropriate entities, or by
>> omitting covariates, or by using the wrong functional form.
>>
>>
>>> I have also seen other empirical approaches like static and dynamic
>> spatial
>>> panel data modelling. I will be reviewing them also to see suitability
>> for
>>> my objective.
>>> But, any further directions or suggestions are highly appreciated.
>>
>> If the data are spatial panel, you can look at the splm package.
>> Personally, I have never found instruments any use at all, because the
>> instruments are typically at best weak because of shared spatial processes
>> with the response, unless the model is really well specified from known
>> theory. In space, almost everything is close to endogeneous unless the
>> opposite is demonstrated. So causal relationships are less worthwhile,
>> because they are at best conditional on omitted variables and
>> autocorrelation engendered by the choice of observational entities.
>>
>> Further, because spatial processes are driven by the inverse matrix of the
>> input graph of proximate neighbours (the covariance matrix of the spatial
>> process), you don't need to start from more than the first order
>> neighbours. Maybe your x has the same spatial pattern as y, so that the
>> residuals are white noise with no spatial structure.
>>
>> Recently, analysts prefer to start with the SLX model (Halleck Vega &
>> Elhorst 2015 and others), so that might be worth exploring. If only the
>> direct impacts seem important, OLS may be enough.
>>
>> Hope this helps,
>>
>> Roger
>>
>>>
>>> Thanks,
>>> -------------------
>>> Lom
>>>
>>>
>>>
>>> On Thu, Jun 4, 2020 at 3:48 AM Roger Bivand <Roger.Bivand using nhh.no> wrote:
>>>
>>>> On Thu, 4 Jun 2020, Lom Navanyo wrote:
>>>>
>>>>> Thank you very much for your support. This gives me what I need and I
>>>> must
>>>>> say listw2sn() is really great.
>>>>>
>>>>> Why do I need the data in the format as in dataout? I am trying to test
>>>>> spatial dependence (or neighborhood effect) by running a regression
>>>>> model that entails pop_size_it = beta_1*sum of pop_size of point i's
>>>>> neighbors within a specified radius. So my plan is to get the neighbors
>>>>> for each focal point as per the specified bands and their attributes
>> (eg
>>>>> pop_size) so I can can add them (attribute) by the bands.
>>>>
>>>> Thanks, clarifies a good deal. Maybe look at the original localG
>> articles
>>>> for exploring distance relationships (Getis and Ord looked at HIV/AIDS);
>>>> ?spdep::localG or
>> https://r-spatial.github.io/spdep/reference/localG.html.
>>>>
>>>> Further note at OLS is biased as you have y = f(y) + e, so y on both
>>>> sides. The nearest equivalent for a single band is
>> spatialreg::lagsarlm()
>>>> with listw=nb2listw(wd1, style="B") to get the neighbour sums through
>> the
>>>> weights matrix. So both your betas and their standard errors are
>> unusable,
>>>> I'm afraid. You are actually very much closer to ordinary kriging,
>> looking
>>>> at the way in which distance attenuates the correlation in value of
>>>> proximate observations.
>>>>
>>>> Hope this clarifies,
>>>>
>>>> Roger
>>>>
>>>>>
>>>>> I am totally new to the area of spatial econometrics, so I am taking
>>>> things
>>>>> one step at a time. Some readings suggest I may need distance matrix or
>>>>> weight matrix but for now I think I should try the current approach.
>>>>>
>>>>> Thank you.
>>>>>
>>>>> -------------
>>>>> Lom
>>>>>
>>>>> On Wed, Jun 3, 2020 at 8:18 AM Roger Bivand <Roger.Bivand using nhh.no>
>> wrote:
>>>>>
>>>>>> On Wed, 3 Jun 2020, Lom Navanyo wrote:
>>>>>>
>>>>>>> I had the errors with rtree using R 3.6.3. I have since changed to R
>>>>>> 4.0.0
>>>>>>> but I got the same error.
>>>>>>>
>>>>>>> And  yes, for Roger's example, I have the objects wd1, ... wd4, all
>>>> with
>>>>>>> length 101. I think my difficulty is my inability to output the list
>>>>>>> detailing the point IDs t50_fid.
>>>>>>
>>>>>> library(spData)
>>>>>> library(sf)
>>>>>> projdata<-st_transform(nz_height, 32759)
>>>>>> pts <- st_coordinates(projdata)
>>>>>> library(spdep)
>>>>>> bufferR <- c(402.336, 1609.34, 3218.69, 4828.03, 6437.38)
>>>>>> bds <- c(0, bufferR)
>>>>>> wd1 <- dnearneigh(pts, bds[1], bds[2])
>>>>>> wd2 <- dnearneigh(pts, bds[2], bds[3])
>>>>>> wd3 <- dnearneigh(pts, bds[3], bds[4])
>>>>>> wd4 <- dnearneigh(pts, bds[4], bds[5])
>>>>>> sn_band1 <- listw2sn(nb2listw(wd1, style="B", zero.policy=TRUE))
>>>>>> sn_band1$band <- paste(attr(wd1, "distances"), collapse="-")
>>>>>> sn_band2 <- listw2sn(nb2listw(wd2, style="B", zero.policy=TRUE))
>>>>>> sn_band2$band <- paste(attr(wd2, "distances"), collapse="-")
>>>>>> sn_band3 <- listw2sn(nb2listw(wd3, style="B", zero.policy=TRUE))
>>>>>> sn_band3$band <- paste(attr(wd3, "distances"), collapse="-")
>>>>>> sn_band4 <- listw2sn(nb2listw(wd4, style="B", zero.policy=TRUE))
>>>>>> sn_band4$band <- paste(attr(wd4, "distances"), collapse="-")
>>>>>> data_out <- do.call("rbind", list(sn_band1, sn_band2, sn_band3,
>>>> sn_band4))
>>>>>> class(data_out) <- "data.frame"
>>>>>> table(data_out$band)
>>>>>> data_out$ID_from <- projdata$t50_fid[data_out$from]
>>>>>> data_out$ID_to <- projdata$t50_fid[data_out$to]
>>>>>> data_out$elev_from <- projdata$elevation[data_out$from]
>>>>>> data_out$elev_to <- projdata$elevation[data_out$to]
>>>>>> str(data_out)
>>>>>>
>>>>>> The "spatial.neighbour" representation was that used in the S-Plus
>>>>>> SpatialStats module, with "from" and "to" columns, and here drops
>>>>>> no-neighbour cases gracefully. So listw2sn() comes in useful
>>>>>> for creating the output, and from there, just look-up in the
>>>>>> input data.frame. Observations here cannot be their own neighbours.
>>>>>>
>>>>>> It would be relevant to know why you need these, are you looking at
>>>>>> variogram clouds?
>>>>>>
>>>>>> Hope this clarifies,
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>>>
>>>>>>> ---------
>>>>>>> Lom
>>>>>>>
>>>>>>> On Tue, Jun 2, 2020 at 8:02 PM Kent Johnson <kent3737 using gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> Roger's example works for me and gives a list of length 101. I did
>>>> have
>>>>>>>> some issues that were resolved by updating packages. I'm using R
>> 3.6.3
>>>>>> on
>>>>>>>> macOS 10.15.4. I also use rtree successfully on Windows 10 with R
>>>> 3.6.3.
>>>>>>>>
>>>>>>>> Kent
>>>>>>>>
>>>>>>>> On Tue, Jun 2, 2020 at 12:29 PM Roger Bivand <Roger.Bivand using nhh.no>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Tue, 2 Jun 2020, Kent Johnson wrote:
>>>>>>>>>
>>>>>>>>>> rtree uses Euclidean distance so the points should be in a
>>>> coordinate
>>>>>>>>>> system where this makes sense at least as a reasonable
>>>> approximation.
>>>>>>>>>
>>>>>>>>> I tried the original example:
>>>>>>>>>
>>>>>>>>> remotes::install_github("hunzikp/rtree")
>>>>>>>>> library(spData)
>>>>>>>>> library(sf)
>>>>>>>>> projdata<-st_transform(nz_height, 32759)
>>>>>>>>> library(rtree)
>>>>>>>>> pts <- st_coordinates(projdata)
>>>>>>>>> rt <- RTree(st_coordinates(projdata))
>>>>>>>>> bufferR <- c(402.336, 1609.34, 3218.69, 4828.03, 6437.38)
>>>>>>>>> wd1 <- withinDistance(rt, pts, bufferR[1])
>>>>>>>>>
>>>>>>>>> but unfortunately failed (maybe newer Boost headers than yours?):
>>>>>>>>>
>>>>>>>>> Error in UseMethod("withinDistance", rTree) :
>>>>>>>>>    no applicable method for 'withinDistance' applied to an object
>> of
>>>>>>>>> class
>>>>>>>>> "c('list', 'RTree')"
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kent
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 2, 2020 at 9:59 AM Roger Bivand <Roger.Bivand using nhh.no>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, 2 Jun 2020, Kent Johnson wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> Date: Tue, 2 Jun 2020 02:44:17 -0500
>>>>>>>>>>>>> From: Lom Navanyo <lomnavasia using gmail.com>
>>>>>>>>>>>>> To: r-sig-geo using r-project.org
>>>>>>>>>>>>> Subject: [R-sig-Geo] How to efficiently generate data of
>>>>>> neighboring
>>>>>>>>>>>>>         points within specified radii (distances) for each
>> point
>>>>>> in a
>>>>>>>>>>> given
>>>>>>>>>>>>>         points data set.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I have data set of about 3400 location points with which I am
>>>>>> trying
>>>>>>>>> to
>>>>>>>>>>>>> generate data of each point and their neighbors within defined
>>>>>> radii
>>>>>>>>>>> (eg,
>>>>>>>>>>>>> 0.25, 1, and 3 miles).
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The rtree package is very fast and memory-efficient for
>>>>>>>>> within-distance
>>>>>>>>>>>> calculations.
>>>>>>>>>>>> https://github.com/hunzikp/rtree
>>>>>>>>>>>
>>>>>>>>>>> Thanks! Does this also apply when the input points are in
>>>>>> geographical
>>>>>>>>>>> coordinates?
>>>>>>>>>>>
>>>>>>>>>>> Roger
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Kent Johnson
>>>>>>>>>>>> Cambridge, MA
>>>>>>>>>>>>
>>>>>>>>>>>>       [[alternative HTML version deleted]]
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> R-sig-Geo mailing list
>>>>>>>>>>>> R-sig-Geo using r-project.org
>>>>>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Roger Bivand
>>>>>>>>>>> Department of Economics, Norwegian School of Economics,
>>>>>>>>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>>>>>>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>>>>>>>>> https://orcid.org/0000-0003-2392-6140
>>>>>>>>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Roger Bivand
>>>>>>>>> Department of Economics, Norwegian School of Economics,
>>>>>>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>>>>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>>>>>>> https://orcid.org/0000-0003-2392-6140
>>>>>>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Roger Bivand
>>>>>> Department of Economics, Norwegian School of Economics,
>>>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>>>> https://orcid.org/0000-0003-2392-6140
>>>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Roger Bivand
>>>> Department of Economics, Norwegian School of Economics,
>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>> https://orcid.org/0000-0003-2392-6140
>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>>>
>>>
>>
>> --
>> Roger Bivand
>> Department of Economics, Norwegian School of Economics,
>> Helleveien 30, N-5045 Bergen, Norway.
>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>> https://orcid.org/0000-0003-2392-6140
>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>
>

-- 
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

More information about the R-sig-Geo mailing list