[R-sig-Geo] Missing local R-squared and residuals in gwr output

Maximilian Sproß Maximilian.Spross at uibk.ac.at
Thu May 10 10:50:17 CEST 2012 

Dear Roger!

Your are right, there are no problems anymore. I did some some 
comparative tests with a small subset of the dataset. The number of 
values in the "pred" column, which are not finite depends on the 
bandwidth. With increasing bandwidth, the NA's disappear.

Unfortunately, I cannot compute gwr.sel due to the large data amount.

By the way, all problems are solved and the use of the cluster is a 
really nice feature to decrease processing time efficiently.

Thank you very much for your help!

Max

On 05/09/2012 01:19 PM, Roger Bivand wrote:
> On Wed, 9 May 2012, Maximilian Sproß wrote:
>
>> Thank you Roger! The gwr on the MPI cluster works fine.
>>
>> However, now the output object includes the intially missing three 
>> data slots: "gwr.e","pred" and "localR2". Unfortunately, the latter 
>> contains only NA's.
>> Sorry for of any inconvenience, but do you think you can solve that?
>
> I do not see any problem there, and indeed it is after the results 
> have been returned from the cluster. You can tell whether you have 
> been into the code block starting on line 261 in spgwr/R/gwr.R if 
> there is no line beginning with "postprocess_localR2" in the timings 
> component of the output object. The conditions are:
>
> ((!fp.given || fit_are_data) && is.null(fittedGWRobject))
>
> where the first is FALSE, the second TRUE and the third TRUE in your 
> case. If the "pred" column in your output contains values that are not 
> finite, this may happen in this code block.
>
> If you cannot see what is going on, we need a smaller test data set 
> that replicates the problem.
>
> Roger
>
>>
>> Thanks in advance and all the best,
>>
>> Max
>>
>>
>> On 05/07/2012 08:45 PM, Roger Bivand wrote:
>>> On Mon, 7 May 2012, "Sproß, Johann" wrote:
>>>
>>>>
>>>>
>>>>
>>>> -- 
>>>> Mag. J. Maximilian Sproß
>>>> Institute of Geography, University of Innsbruck
>>>> Innrain 52
>>>> A-6020 INNSBRUCK
>>>>
>>>> Tel. +43 (0)512 507 5413
>>>> web: http://www.uibk.ac.at/geographie/projects/lidar/
>>>>
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Roger Bivand [mailto:Roger.Bivand at nhh.no]
>>>> Gesendet: Mo 07.05.2012 14:48
>>>> An: Maximilian Sproß
>>>> Cc: r-sig-geo
>>>> Betreff: Re: [R-sig-Geo] Missing local R-squared and residuals in 
>>>> gwr output
>>>>
>>>> On Mon, 7 May 2012, Maximilian Sproß wrote:
>>>>
>>>>> Dear Roger!
>>>>>
>>>>> Thank you very much for your fast reply and work!
>>>>>
>>>>> I'm not really an expert in HPC-computing, but i will try to 
>>>>> report as goog
>>>>> as i can.
>>>>>
>>>>> I updated spgwr and started a job on the cluster which takes 
>>>>> normally 1,5 h.
>>>>> So far, it run for 5 hours, which indicates that the 
>>>>> parallelization does not
>>>>> work efficient anymore. The function makeCluster(64, type="MPI") 
>>>>> worked fine.
>>>>> Our cluster runs with openMPI.
>>>>
>>>> Correct. I'll try to add back an option to use snow instead of 
>>>> parallel.
>>>>
>>>> I tried out the new version but it seems still using parallel.
>>>>
>>>> code:
>>>>
>>>> gwr_50 <- 
>>>> gwr(hef at data$DIF~hef at data$ELEVATION+hef at data$SKY+hef at data$SLOPE+hef at data$SOLAR+factor(asp_fac), 
>>>> data=hef, bandwidth=50, gweight=gwr.Gauss,fit.points=coords, 
>>>> hatmatrix=FALSE, cl=cl)
>>>
>>> Add use_snow=TRUE to the command to switch to snow.
>>>
>>> Roger
>>>
>>>> Loading required package: parallel
>>>>
>>>> Attaching package: 'parallel'
>>>>
>>>> The following object(s) are masked from 'package:snow':
>>>>
>>>>    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
>>>>    clusterExport, clusterMap, clusterSplit, makeCluster, parApply,
>>>>    parCapply, parLapply, parRapply, parSapply, splitIndices,
>>>>    stopCluster
>>>>
>>>> Max
>>>>
>>>>
>>>> When it reaches R-forge, its revision number will be > 1252.
>>>>
>>>> Roger
>>>>
>>>>>
>>>>> In that context, i found on the CRAN Task view: High-Performance 
>>>>> and Parallel
>>>>> Computing with R the following:
>>>>> "<http://www.dict.cc/englisch-deutsch/parallelization.html>Direct 
>>>>> support in
>>>>> R is starting with release 2.14.0 which includes a new package 
>>>>> parallel
>>>>> incorporating (slightly revised) copies of packages multicore and 
>>>>> snow (*but
>>>>> excluding MPI, PVM and NWS clusters*). Does the new parallel 
>>>>> support works
>>>>> still in the openMPI environment?
>>>>>
>>>>> regards,
>>>>>
>>>>> Max
>>>>>
>>>>> fyi:
>>>>>
>>>>> sessionInfo()
>>>>> R version 2.14.0 (2011-10-31)
>>>>> Platform: x86_64-unknown-linux-gnu (64-bit)
>>>>>
>>>>> locale:
>>>>> [1] LC_CTYPE=en_US       LC_NUMERIC=C         LC_TIME=en_US
>>>>> [4] LC_COLLATE=en_US     LC_MONETARY=en_US    LC_MESSAGES=en_US
>>>>> [7] LC_PAPER=C           LC_NAME=C            LC_ADDRESS=C
>>>>> [10] LC_TELEPHONE=C       LC_MEASUREMENT=en_US LC_IDENTIFICATION=C
>>>>>
>>>>> attached base packages:
>>>>> [1] parallel  stats     graphics  grDevices utils     datasets  
>>>>> methods
>>>>> [8] base
>>>>>
>>>>> other attached packages:
>>>>> [1] spgwr_0.6-15    spdep_0.5-45    coda_0.14-6     deldir_0.0-16
>>>>> [5] maptools_0.8-10 foreign_0.8-46  nlme_3.1-102    MASS_7.3-16
>>>>> [9] Matrix_1.0-1    lattice_0.20-0  boot_1.3-3      gstat_1.0-10
>>>>> [13] spacetime_0.5-7 xts_0.8-2       zoo_1.7-6       sp_0.9-98
>>>>> [17] snow_0.3-8      Rmpi_0.5-9
>>>>>
>>>>> loaded via a namespace (and not attached):
>>>>> [1] grid_2.14.0
>>>>>
>>>>>
>>>>> On 05/05/2012 04:24 PM, Roger Bivand wrote:
>>>>>> On Fri, 4 May 2012, Maximilian Sproß wrote:
>>>>>>
>>>>>>> Dear r-sig-geo list!
>>>>>>>
>>>>>>> I run gwr on a multi-node cluster(on 64 slots). In the gwr 
>>>>>>> output (slot
>>>>>>> "SDF"), the gwr residuals and the local R-squared are missing. When
>>>>>>> performing the same model on the local machine, these components 
>>>>>>> are
>>>>>>> included. Unfortunately, the calculation in this way takes about 
>>>>>>> 5 days
>>>>>>> instead of few hours when using the cluster.
>>>>>>>
>>>>>>> Perhaps, that problem arises due to the argument "fit.points", 
>>>>>>> which has
>>>>>>> to be passed if the local coefficient estimates should be made on a
>>>>>>> multi node cluster.
>>>>>>>
>>>>>>> Does anyone have an idea how to solve that problem with the missing
>>>>>>> local R-squared and residuals if the gwr is calculated on a 
>>>>>>> cluster?
>>>>>>
>>>>>> The understanding for use on a cluster was that the data points 
>>>>>> and the fit
>>>>>> points are different, so there is no observed dependent variable 
>>>>>> at the fit
>>>>>> point, hence no local R2. I've added logic in the code that 
>>>>>> checks for
>>>>>> equality between the fit and data points, and this for me 
>>>>>> resolves the
>>>>>> problem, but may break other things. I've committed to R-forge, 
>>>>>> project
>>>>>> rspatial, module spgwr. The source tarball and binary packages 
>>>>>> should be
>>>>>> available later this evening European time from:
>>>>>>
>>>>>> https://r-forge.r-project.org/R/?group_id=1014
>>>>>>
>>>>>> Could you please try it out, and report back? I should also 
>>>>>> migrate spgwr
>>>>>> from snow to parallel before I release it.
>>>>>>
>>>>>> Best wishes,
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thank you very much in advance!
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> Max
>>>>>>>
>>>>>>>
>>>>>>> selected R-code:
>>>>>>>
>>>>>>> ### gwr on local machine:
>>>>>>>
>>>>>>> gwr_50 <-
>>>>>>> gwr(hef at data$DIF~hef at data$ELEVATION+hef at data$SKY+hef at data$SLOPE+hef at data$SOLAR, 
>>>>>>> data=hef, bandwidth=50, gweight=gwr.Gauss)
>>>>>>>
>>>>>>>
>>>>>>> # part of the  str(gwr_50) output...
>>>>>>>
>>>>>>>
>>>>>>> List of 11
>>>>>>>  $ SDF      :Formal class 'SpatialPointsDataFrame' [package 
>>>>>>> "sp"] with 5
>>>>>>> slots
>>>>>>>   .. ..@ data       :'data.frame':    286288 obs. of  9 variables:
>>>>>>>   .. .. ..$ sum.w      : num [1:286288] 2009 2003 2091 2089 2086 
>>>>>>> ...
>>>>>>>   .. .. ..$ (Intercept): num [1:286288] -28.7 -28.5 -29.9 -29.7 
>>>>>>> -29.5 ...
>>>>>>>   .. .. ..$ elevation  : num [1:286288] 0.0139 0.0138 0.014 
>>>>>>> 0.014 0.014
>>>>>>> ...
>>>>>>>   .. .. ..$ sky        : num [1:286288] -0.153 -0.155 -0.146 
>>>>>>> -0.148 -0.149
>>>>>>> ...
>>>>>>>   .. .. ..$ slope      : num [1:286288] -2.58 -2.61 -2.42 -2.45 
>>>>>>> -2.48 ...
>>>>>>>   .. .. ..$ solar      : num [1:286288] -0.00139 -0.00136 
>>>>>>> -0.0015 -0.00147
>>>>>>> -0.00144 ...
>>>>>>>   .. .. ..$ gwr.e      : num [1:286288] -0.461 -0.683 -0.5987 
>>>>>>> -0.2692
>>>>>>> 0.0406 ...
>>>>>>>   .. .. ..$ pred       : num [1:286288] 0.806 0.833 0.507 0.514 
>>>>>>> 0.576 ...
>>>>>>>   .. .. ..$ localR2    : num [1:286288] 0.621 0.618 0.638 0.635 
>>>>>>> 0.632 ...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ### gwr on cluster :
>>>>>>>
>>>>>>> cl <- makeCluster(32, type="MPI")
>>>>>>>
>>>>>>> coords <- coordinates(hef)
>>>>>>>
>>>>>>> gw <-
>>>>>>> gwr(hef at data$DIF~hef at data$ELEVATION+hef at data$SKY+hef at data$SLOPE+hef at data$SOLAR, 
>>>>>>> data=hef, bandwidth=50, gweight=gwr.Gauss,fit.points=coords,
>>>>>>> hatmatrix=FALSE, cl=cl)
>>>>>>>
>>>>>>> # part of the  str(gwr_50) output...
>>>>>>>
>>>>>>> List of 11
>>>>>>>  $ SDF      :Formal class 'SpatialPointsDataFrame' [package 
>>>>>>> "sp"] with 5
>>>>>>> slots
>>>>>>>   .. ..@ data       :'data.frame':    286288 obs. of  6 variables:
>>>>>>>   .. .. ..$ sum.w      : num [1:286288] 1 1 1 1 1 ...
>>>>>>>   .. .. ..$ (Intercept): num [1:286288] 12541 1970 2057 -1505 
>>>>>>> -1030 ...
>>>>>>>   .. .. ..$ elevation  : num [1:286288] -3.891 -0.602 -0.738 
>>>>>>> 0.465 0.309
>>>>>>> ...
>>>>>>>   .. .. ..$ sky        : num [1:286288] -0.954 -0.425 3.714 
>>>>>>> 0.159 0.152
>>>>>>> ...
>>>>>>>   .. .. ..$ slope      : num [1:286288] 62.19 NA -27.21 1.95 
>>>>>>> 16.03 ...
>>>>>>>   .. .. ..$ solar      : num [1:286288] NA NA NA NA 0.042 ...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>     [[alternative HTML version deleted]]
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> R-sig-Geo mailing list
>>>>>>> R-sig-Geo at r-project.org
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> -- 
>>>> Roger Bivand
>>>> Department of Economics, NHH Norwegian School of Economics,
>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>> voice: +47 55 95 93 55; fax +47 55 95 95 43
>>>> e-mail: Roger.Bivand at nhh.no
>>>>
>>>>
>>>
>>
>>
>

More information about the R-sig-Geo mailing list