[R-sig-Geo] Creating very large spatial weight matrix

[Michael Sumner]

Sorry, I also realize that your vector is just too long - R is limited
to a 32-bit index, so it just cannot be done without special tricks in
separate vectors.

The ff package documentation mentions that 64-bit indexing may be
included in a future release, and there have been discussions on this
for base R. (perhaps for R 3.0)

Cheers, Mike


On Fri, Nov 19, 2010 at 10:40 AM, Michael Sumner <mdsumner at gmail.com> wrote:
> And, please report your OS and version of R (64-bit presumably?).
>
>
>
> On Fri, Nov 19, 2010 at 10:39 AM, Michael Sumner <mdsumner at gmail.com> wrote:
>> In general you need at least twice the required memory, and it has to
>> be contiguous. Try with a fresh instance of R and try to create a
>> single vector of that size, that might show that you *could* do it.
>>
>> Otherwise, check out the ff package, and see other options in the High
>> Performance Computing Task View on CRAN.
>>
>> There may be other techniques you can use to solve the problem, but
>> those two things are my direct answers to your questions.
>>
>> Cheers, Mike.
>>
>> On Fri, Nov 19, 2010 at 10:28 AM, Aleksandr Andreev
>> <aleksandr.andreev at gmail.com> wrote:
>>> Hello list,
>>>
>>> I have 120,000 geocoded observations, for which I'm trying to create a
>>> distance-based spatial weighting matrix so that I can perform a Moran
>>> test.
>>>
>>> Each observation has Lat and Lon.
>>>
>>> Unfortunately, when I run
>>> dists <- as.matrix(dist(cbind(Lon, Lat)))
>>> I get the message:
>>> Error in vector("double", length) : vector size specified is too large
>>>
>>> Now I realize that 120,000^2 / 2 is on the order of 6 GB. However, I
>>> seem to be running into software limitations on the vector size before
>>> I hit RAM limitations. Also, in principle, it should be possible
>>> (though slow) to use hard disk space to store this matrix. Does anyone
>>> have any ideas on how to do this in R?
>>>
>>> Thanks,
>>>
>>> ------------------------
>>> Aleksandr Andreev
>>> Graduate Student - Department of Economics
>>> University of North Carolina at Chapel Hill
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at stat.math.ethz.ch
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>>
>>
>> --
>> Michael Sumner
>> Institute for Marine and Antarctic Studies, University of Tasmania
>> Hobart, Australia
>> e-mail: mdsumner at gmail.com
>>
>
>
>
> --
> Michael Sumner
> Institute for Marine and Antarctic Studies, University of Tasmania
> Hobart, Australia
> e-mail: mdsumner at gmail.com
>



-- 
Michael Sumner
Institute for Marine and Antarctic Studies, University of Tasmania
Hobart, Australia
e-mail: mdsumner at gmail.com