[R-sig-Geo] Creating very large spatial weight matrix

[Aleksandr Andreev]

Yes, sorry, I'm running R 2.12.0 on Ubuntu 64-bit (kernel 2.6.32-25-generic)

Thanks for pointing out ff.


------------------------
Aleksandr Andreev
Graduate Student - Department of Economics
University of North Carolina at Chapel Hill
Mobile: +1 303 507 93 88
Skype: typiconman



2010/11/18 Michael Sumner <mdsumner at gmail.com>:
> And, please report your OS and version of R (64-bit presumably?).
>
>
>
> On Fri, Nov 19, 2010 at 10:39 AM, Michael Sumner <mdsumner at gmail.com> wrote:
>> In general you need at least twice the required memory, and it has to
>> be contiguous. Try with a fresh instance of R and try to create a
>> single vector of that size, that might show that you *could* do it.
>>
>> Otherwise, check out the ff package, and see other options in the High
>> Performance Computing Task View on CRAN.
>>
>> There may be other techniques you can use to solve the problem, but
>> those two things are my direct answers to your questions.
>>
>> Cheers, Mike.
>>
>> On Fri, Nov 19, 2010 at 10:28 AM, Aleksandr Andreev
>> <aleksandr.andreev at gmail.com> wrote:
>>> Hello list,
>>>
>>> I have 120,000 geocoded observations, for which I'm trying to create a
>>> distance-based spatial weighting matrix so that I can perform a Moran
>>> test.
>>>
>>> Each observation has Lat and Lon.
>>>
>>> Unfortunately, when I run
>>> dists <- as.matrix(dist(cbind(Lon, Lat)))
>>> I get the message:
>>> Error in vector("double", length) : vector size specified is too large
>>>
>>> Now I realize that 120,000^2 / 2 is on the order of 6 GB. However, I
>>> seem to be running into software limitations on the vector size before
>>> I hit RAM limitations. Also, in principle, it should be possible
>>> (though slow) to use hard disk space to store this matrix. Does anyone
>>> have any ideas on how to do this in R?
>>>
>>> Thanks,
>>>
>>> ------------------------
>>> Aleksandr Andreev
>>> Graduate Student - Department of Economics
>>> University of North Carolina at Chapel Hill
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at stat.math.ethz.ch
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>>
>>
>> --
>> Michael Sumner
>> Institute for Marine and Antarctic Studies, University of Tasmania
>> Hobart, Australia
>> e-mail: mdsumner at gmail.com
>>
>
>
>
> --
> Michael Sumner
> Institute for Marine and Antarctic Studies, University of Tasmania
> Hobart, Australia
> e-mail: mdsumner at gmail.com
>