[R-sig-Geo] strange issue of memory converting a raster to a data frame

Tue Jan 24 15:27:54 CET 2023

Dear Vlad and Loic,

Some more info on the issue; the error ("Error: cannot allocate vector of
size 1.7 Gb") shows a random behaviour, sometimes things work (e.g., after
a reboot) and sometimes not.

 Anyway this is a reproducible example:

theTest=rast(nrows=5000, ncols=5000, nlyrs=18, xmin=0, ymin=0,
vals=3333.333)
mem_info(theTest)
theDataFrame=as.data.frame(theTest)

Sebastiano

On Mon, Jan 23, 2023 at 7:18 PM Sebastiano Trevisani <strevisani using iuav.it>
wrote:

> Thank you Vlad!
> This is a good solution, I keep it for future.
>
> My issue is that I would like to understand why things don't work in the
> best performing PC.
>
> I'll do more experiments, to isolate the issue.
>
> Sebastiano
>
>
>
> Sebastiano
>
>
>
> Il lun 23 gen 2023, 18:48 Vlad Amihaesei <vlad.amihaesei95 using gmail.com> ha
> scritto:
>
>> You can try converting each layer to data frame instead of all the layers
>> at once time. You can do this using a loop.
>>
>> r <- terra::rast(file.nc)
>>
>> df<- NULL
>>
>> for( i in 1:nlyr(r)){
>> r1 <-  r[[i]]
>> r1.df <- as.data.frame(r1,xy =T)
>> df <- rbind(df, r1.df)
>> }
>>
>> Best  regards
>>
>> Vlad Alexandru AMIHAESEI
>>
>>
>> On Mon, Jan 23, 2023 at 10:33 AM Sebastiano Trevisani <strevisani using iuav.it>
>> wrote:
>>
>>> Hi  Loïc,
>>>
>>> thank you for your suggestion! this was one of the possibilities I
>>> checked.
>>> However, I tried with terra.options() to change the memfrac parameter
>>> from
>>> 0.6 to 0.1, so as to force the writing on disc, but things didn't work.
>>> It seems like it is working as 32 bit system, but the R the installation
>>> seems right to 64. Probably I'm missing something very obvious.
>>> I will try to build a reproducible example without the need to upload all
>>> the file.
>>>
>>> Thanks again,
>>>
>>> Sebastiano
>>>
>>> *      Sebastiano Trevisani, Ph.D.*
>>> *            Associate Professor*
>>> *Applied and Environmental Geology*
>>>
>>>
>>> *  https://orcid.org/0000-0001-8436-7798
>>> <https://orcid.org/0000-0001-8436-7798>*
>>> *IUAV University of Venice: www.iuav.it <http://www.iuav.it/>*
>>>
>>> *Address: Dorsoduro 2206,  Venice 30123, Italy Tel:    +39. 041. 257
>>> 1299Mail:    strevisani using iuav.it <strevisani using iuav.it> *
>>> *"Le opinioni espresse sono riferibili esclusivamente all'autore e non *
>>> * riflettono in alcun modo una posizione ufficiale dello IUAV "*
>>> *"The views expressed are purely those of the writer and may not in
>>> any circumstances be regarded as stating an official position of the
>>> IUAV."*
>>>
>>>
>>> On Mon, Jan 23, 2023 at 5:36 PM DUTRIEUX Loic <
>>> Loic.DUTRIEUX using ec.europa.eu>
>>> wrote:
>>>
>>> > Hi Sebastiano,
>>> >
>>> > Try comparing terra::terraOptions() and terra::mem_info(r) on both
>>> > systems. Just guessing there but it could be that terra is trying to do
>>> > everything in memory and failing on your 16GB system, while taking a
>>> memory
>>> > friendly route on your 8GB system.
>>> >
>>> > Cheers,
>>> > Loïc
>>> > ________________________________________
>>> > From: R-sig-Geo <r-sig-geo-bounces using r-project.org> on behalf of
>>> Sebastiano
>>> > Trevisani <strevisani using iuav.it>
>>> > Sent: 23 January 2023 12:30:14
>>> > To: r-sig-geo
>>> > Subject: [R-sig-Geo] strange issue of memory converting a raster to a
>>> data
>>> > frame
>>> >
>>> > Dear list members,
>>> >
>>> > In converting a raster as the one below to a data frame, I have an
>>> issue of
>>> > memory allocation only when I use a specific pc, and in particular the
>>> one
>>> > with more memory. To be more precise (using same version of R 4.2.2 and
>>> > Terra terra 1.6.47, by means of as.data.frame() function):
>>> >
>>> > with a I5 processor with 8Gb of memory, windows 10 it works
>>> >
>>> > with a I7 processor with 16gb of memory, windows 11 it doesn't
>>> >
>>> > I'm not able to figure out a possible explanation!
>>> > Any suggestions on what to check?
>>> >
>>> >
>>> > SpatRaster
>>> > dimensions  : 5000, 5000, 18  (nrow, ncol, nlyr)
>>> > resolution  : 30, 30  (x, y)
>>> > extent      : -8526.878, 141473.1, 4110588, 4260588  (xmin, xmax, ymin,
>>> > ymax)
>>> > coord. ref. : WGS 84 / UTM zone 45N (EPSG:32645)
>>> > source      : multyMadRes2c.tif
>>> >
>>> >
>>> >
>>> > Sebastiano
>>> >
>>> > *      Sebastiano Trevisani, Ph.D.*
>>> > *            Associate Professor*
>>> > *Applied and Environmental Geology*
>>> >
>>> >
>>> > *
>>> >
>>> https://urldefense.com/v3/__https://orcid.org/0000-0001-8436-7798__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9YxpzJmj$
>>> > <
>>> >
>>> https://urldefense.com/v3/__https://orcid.org/0000-0001-8436-7798__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9YxpzJmj$
>>> > >*
>>> > *IUAV University of Venice:
>>> >
>>> https://urldefense.com/v3/__http://www.iuav.it__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9ag_1m0F$
>>> > <
>>> >
>>> https://urldefense.com/v3/__http://www.iuav.it/__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9cV-rY9s$
>>> > >*
>>> >
>>> > *Address: Dorsoduro 2206,  Venice 30123, Italy Tel:    +39. 041. 257
>>> > 1299Mail:    strevisani using iuav.it <strevisani using iuav.it> *
>>> > *"Le opinioni espresse sono riferibili esclusivamente all'autore e non
>>> *
>>> > * riflettono in alcun modo una posizione ufficiale dello IUAV "*
>>> > *"The views expressed are purely those of the writer and may not in
>>> > any circumstances be regarded as stating an official position of the
>>> > IUAV."*
>>> >
>>> >         [[alternative HTML version deleted]]
>>> >
>>> > _______________________________________________
>>> > R-sig-Geo mailing list
>>> > R-sig-Geo using r-project.org
>>> >
>>> >
>>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-geo__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9ScsuXEI$
>>> >
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>

	[[alternative HTML version deleted]]