[R-sig-Geo] Large NetCDF file problems

Wed Jun 18 17:19:09 CEST 2014

Pablo:

Although probably not ideal, my general approach when dealing with
NetCDFs is first extract/convert them to a more friendly format --
gdalUtils should work on a Windows box to accomplish this (let me know
if it doesn't).

--j

On Thu, Jun 12, 2014 at 2:39 AM, Michael Sumner <mdsumner at gmail.com> wrote:
> My take is that it's an outdated NetCDF library being shipped with RNetCDF
> and ncdf. Here's a minimal reproducible example that behaves fine in 32-bit
> R, but not 64-bit. Reduce n to 2^14 and it works in either:
>
> n <- 2^15
>
> library(ncdf)
> d1 <- dim.def.ncdf("x", "count", 1:n)
> d2 <- dim.def.ncdf("y", "count", 1:n)
> v <- var.def.ncdf("v", "count", list(d1, d2), -1)
> nc <- create.ncdf("afile.nc", v)
>
>
> You won't see this in Linux installs because they will usually be using
> later versions of NetCDF. (It's a special CRAN service to build these
> packages on a Windows machine with NetCDF installed.)
>
> I am interested in learning more about compiling for myself in Windows, but
> it's just quite painful.
>
> HTH
>
>
> On Thu, Jun 12, 2014 at 10:45 AM, Pablo Alfaro <palfaro at motionsoft.com.uy>
> wrote:
>
>> So you think this is not an R problem then?
>> The executable failing the assertion is rsession.exe
>>
>>
>> 2014-06-11 18:14 GMT-03:00 Roger Bivand <Roger.Bivand at nhh.no>:
>>
>> I suggest that you put together a complete reproducible example with
>>> access to the file on a server, and send this to the maintainers of ncdf
>>> and RNetCDF. This looks either like an issue in compiler flags, or an issue
>>> in the code of the external library used.
>>>
>>> Roger
>>>
>>>
>>> On Wed, 11 Jun 2014, Pablo Alfaro wrote:
>>>
>>>  Hmm, just tried using RNetCDF (v 1.6.2-2) and it has the same problem.
>>>>
>>>> When using
>>>>
>>>> require('RNetCDF')
>>>> nc <- open.nc('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%3A00')
>>>> var.get.nc(nc, "T2", start=c(1, 1, 1), count=c(NA, NA, NA))
>>>>
>>>> I get the same problem on 64bit R, no problem on 32bits.
>>>>
>>>> Seems like a problem with the version of posixio.c used in 64 bit R. Just
>>>> found a version of posixio.c on the web (
>>>> http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/util/sorc/
>>>> ncdump.fd/libsrc/posixio.c)
>>>> which has the assertion on line 417.
>>>>
>>>> assert(offset >= 0);
>>>>
>>>> offset is a variable of type off_t which is defined in <sys/types.h> and
>>>> apparently requires that you enable Large File Support on the compiler
>>>> command line to be a 64 bit integer.
>>>> Can it be that the 32 bit version of R is using Large File Support but
>>>> the
>>>> 64 bit version is not?
>>>>
>>>> Best
>>>> Pablo
>>>>
>>>>
>>>>
>>>>
>>>> 2014-06-11 15:07 GMT-03:00 Pablo Alfaro <palfaro at motionsoft.com.uy>:
>>>>
>>>>  Hi Michael, Pascal, thank you both for your replies.
>>>>>
>>>>> Yeah, Sorry, I'm using R 3.0.3 64 (or 32 for the working version) bit,
>>>>> on
>>>>> Windows 7 x64 and the NetCDF library is ncdf (not ncdf4) version 1.6.6,
>>>>> the
>>>>> last available version.
>>>>>
>>>>> I need it to run on Windows since the servers I run the code on are
>>>>> Windows.
>>>>>
>>>>> I tried downloading and installing the package you mention Michael but
>>>>> when loading it, it says the x64 architecture is not supported (the
>>>>> author
>>>>> mentions it in the site as well). The thing is that ncdf works on x86
>>>>> so it
>>>>> leaves me with the same options.
>>>>>
>>>>> RNetCDF seems to use NetCDF v3, but maybe it doesn't have this problem.
>>>>> I
>>>>> wil try that and see if it helps.
>>>>>
>>>>> The assertion on the error complains an offset which isn't >= 0. If I
>>>>> were
>>>>> to take a guess, that offset is a 32 bit signed integer and that's why
>>>>> it
>>>>> breaks only with large files (bigger than 2^31 - 1 bytes). But I can't
>>>>> tell
>>>>> for sure. Also it's weird that it works on 32 bits.
>>>>>
>>>>> Best
>>>>> Pablo
>>>>>
>>>>>
>>>>> 2014-06-10 23:13 GMT-03:00 Michael Sumner <mdsumner at gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Wed, Jun 11, 2014 at 11:49 AM, Pascal Oettli <kridox at ymail.com>
>>>>>> wrote:
>>>>>>
>>>>>>  Hello,
>>>>>>>
>>>>>>> Do you get the same error if you run your code out of Rstudio?
>>>>>>>
>>>>>>> However, another approach would be to open your NetCDF file with
>>>>>>> 'brick' function from the raster package.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I don't think either of these will help, raster simply runs ncdf (or
>>>>>> preferably ncdf4) to do the file read. You might find it easier to
>>>>>> move to
>>>>>> ncdf4 if you can, but potentially the stuff I write below is just an
>>>>>> unwelcome aside. I have seen a similar error in a file I've tried to
>>>>>> use,
>>>>>> so I'll have another look at that and see if I can isolate where the
>>>>>> problem lies. At minimum we really need to know the version of the
>>>>>> NetCDF
>>>>>> library in use and perhaps details of how it was compiled so at
>>>>>> minimum you
>>>>>> should provide the ncdf package version as well as R's.
>>>>>>
>>>>>> It's possible to install ncdf4 on Windows, but it's not supported (no
>>>>>> binary .zip on CRAN) but you can do it yourself (again not supported,
>>>>>> and
>>>>>> the archive here is not updated):
>>>>>> http://cirrus.ucsd.edu/~pierce/ncdf/
>>>>>>
>>>>>> It's much more generally available to do this on Linux, though there's
>>>>>> always bumps in the road.
>>>>>>
>>>>>> RNetCDF is actively updated on CRAN and is another option to see if the
>>>>>> read is supported (I don't know how to find out what version of the
>>>>>> library
>>>>>> was used for the CRAN Windows binary, but presambly it is < 4.0).
>>>>>>
>>>>>> Cheers, Mike.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  Regards,
>>>>>>> Pascal
>>>>>>>
>>>>>>> On Wed, Jun 11, 2014 at 9:10 AM, Pablo Alfaro <
>>>>>>> palfaro at motionsoft.com.uy>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, I'm having trouble opening a large netcdf file (2.45 GB) with the
>>>>>>>> following code on R 3.0.3 64 bit
>>>>>>>>
>>>>>>>> require('ncdf')
>>>>>>>> nc <- open.ncdf('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%
>>>>>>>> 3A00')
>>>>>>>> get.var.ncdf(nc, 'T2', start=c(1, 1, 1), count=c(-1, -1, -1))
>>>>>>>>
>>>>>>>> I get this error:
>>>>>>>> Assertion failed!
>>>>>>>>
>>>>>>>> Program: C:\Program Files\RStudio\bin\x64\rsession.exe
>>>>>>>> File: posixio.c, Line 417
>>>>>>>>
>>>>>>>> Expression: offset >= 0
>>>>>>>>
>>>>>>>> This application has requested the Runtime to terminate it in an
>>>>>>>>
>>>>>>> unusual
>>>>>>>
>>>>>>>> way.
>>>>>>>> Please contact the application's support team for more information.
>>>>>>>>
>>>>>>>>
>>>>>>>> In R 32 bit there are no problems.
>>>>>>>>
>>>>>>>> It is the same problem as reported here:
>>>>>>>>
>>>>>>>>  http://r.789695.n4.nabble.com/Problems-with-Raster-and-clim-
>>>>>>> pact-packages-with-large-netcdf-files-2-7G-in-x64-bit-
>>>>>>> R-td4119638.html#a4691956
>>>>>>>
>>>>>>>>
>>>>>>>> Does anyone have any clue as to why this may be?
>>>>>>>> I can upload the file on an FTP server but it'll take me a while.
>>>>>>>>
>>>>>>>> Thanks in advance!
>>>>>>>> Pablo
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ing. Pablo Alfaro Pi??eyro
>>>>>>>>
>>>>>>>> MotionSoft Consulting S.R.L.
>>>>>>>> Tel: 095 938 487
>>>>>>>>
>>>>>>>>         [[alternative HTML version deleted]]
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> R-sig-Geo mailing list
>>>>>>>> R-sig-Geo at r-project.org
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pascal Oettli
>>>>>>> Project Scientist
>>>>>>> JAMSTEC
>>>>>>> Yokohama, Japan
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> R-sig-Geo mailing list
>>>>>>> R-sig-Geo at r-project.org
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Michael Sumner
>>>>>> Software and Database Engineer
>>>>>> Australian Antarctic Division
>>>>>> Hobart, Australia
>>>>>> e-mail: mdsumner at gmail.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ing. Pablo Alfaro Pi??eyro
>>>>>
>>>>> MotionSoft Consulting S.R.L.
>>>>> Tel: 095 938 487
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Roger Bivand
>>> Department of Economics, Norwegian School of Economics,
>>> Helleveien 30, N-5045 Bergen, Norway.
>>> voice: +47 55 95 93 55; fax +47 55 95 91 00
>>> e-mail: Roger.Bivand at nhh.no
>>>
>>>
>>
>>
>> --
>> Ing. Pablo Alfaro Piñeyro
>>
>> MotionSoft Consulting S.R.L.
>> Tel: 095 938 487
>>
>>
>> 2014-06-11 18:14 GMT-03:00 Roger Bivand <Roger.Bivand at nhh.no>:
>>
>> I suggest that you put together a complete reproducible example with
>>> access to the file on a server, and send this to the maintainers of ncdf
>>> and RNetCDF. This looks either like an issue in compiler flags, or an issue
>>> in the code of the external library used.
>>>
>>> Roger
>>>
>>>
>>> On Wed, 11 Jun 2014, Pablo Alfaro wrote:
>>>
>>>  Hmm, just tried using RNetCDF (v 1.6.2-2) and it has the same problem.
>>>>
>>>> When using
>>>>
>>>> require('RNetCDF')
>>>> nc <- open.nc('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%3A00')
>>>> var.get.nc(nc, "T2", start=c(1, 1, 1), count=c(NA, NA, NA))
>>>>
>>>> I get the same problem on 64bit R, no problem on 32bits.
>>>>
>>>> Seems like a problem with the version of posixio.c used in 64 bit R. Just
>>>> found a version of posixio.c on the web (
>>>> http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/util/sorc/
>>>> ncdump.fd/libsrc/posixio.c)
>>>> which has the assertion on line 417.
>>>>
>>>> assert(offset >= 0);
>>>>
>>>> offset is a variable of type off_t which is defined in <sys/types.h> and
>>>> apparently requires that you enable Large File Support on the compiler
>>>> command line to be a 64 bit integer.
>>>> Can it be that the 32 bit version of R is using Large File Support but
>>>> the
>>>> 64 bit version is not?
>>>>
>>>> Best
>>>> Pablo
>>>>
>>>>
>>>>
>>>>
>>>> 2014-06-11 15:07 GMT-03:00 Pablo Alfaro <palfaro at motionsoft.com.uy>:
>>>>
>>>>  Hi Michael, Pascal, thank you both for your replies.
>>>>>
>>>>> Yeah, Sorry, I'm using R 3.0.3 64 (or 32 for the working version) bit,
>>>>> on
>>>>> Windows 7 x64 and the NetCDF library is ncdf (not ncdf4) version 1.6.6,
>>>>> the
>>>>> last available version.
>>>>>
>>>>> I need it to run on Windows since the servers I run the code on are
>>>>> Windows.
>>>>>
>>>>> I tried downloading and installing the package you mention Michael but
>>>>> when loading it, it says the x64 architecture is not supported (the
>>>>> author
>>>>> mentions it in the site as well). The thing is that ncdf works on x86
>>>>> so it
>>>>> leaves me with the same options.
>>>>>
>>>>> RNetCDF seems to use NetCDF v3, but maybe it doesn't have this problem.
>>>>> I
>>>>> wil try that and see if it helps.
>>>>>
>>>>> The assertion on the error complains an offset which isn't >= 0. If I
>>>>> were
>>>>> to take a guess, that offset is a 32 bit signed integer and that's why
>>>>> it
>>>>> breaks only with large files (bigger than 2^31 - 1 bytes). But I can't
>>>>> tell
>>>>> for sure. Also it's weird that it works on 32 bits.
>>>>>
>>>>> Best
>>>>> Pablo
>>>>>
>>>>>
>>>>> 2014-06-10 23:13 GMT-03:00 Michael Sumner <mdsumner at gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Wed, Jun 11, 2014 at 11:49 AM, Pascal Oettli <kridox at ymail.com>
>>>>>> wrote:
>>>>>>
>>>>>>  Hello,
>>>>>>>
>>>>>>> Do you get the same error if you run your code out of Rstudio?
>>>>>>>
>>>>>>> However, another approach would be to open your NetCDF file with
>>>>>>> 'brick' function from the raster package.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I don't think either of these will help, raster simply runs ncdf (or
>>>>>> preferably ncdf4) to do the file read. You might find it easier to
>>>>>> move to
>>>>>> ncdf4 if you can, but potentially the stuff I write below is just an
>>>>>> unwelcome aside. I have seen a similar error in a file I've tried to
>>>>>> use,
>>>>>> so I'll have another look at that and see if I can isolate where the
>>>>>> problem lies. At minimum we really need to know the version of the
>>>>>> NetCDF
>>>>>> library in use and perhaps details of how it was compiled so at
>>>>>> minimum you
>>>>>> should provide the ncdf package version as well as R's.
>>>>>>
>>>>>> It's possible to install ncdf4 on Windows, but it's not supported (no
>>>>>> binary .zip on CRAN) but you can do it yourself (again not supported,
>>>>>> and
>>>>>> the archive here is not updated):
>>>>>> http://cirrus.ucsd.edu/~pierce/ncdf/
>>>>>>
>>>>>> It's much more generally available to do this on Linux, though there's
>>>>>> always bumps in the road.
>>>>>>
>>>>>> RNetCDF is actively updated on CRAN and is another option to see if the
>>>>>> read is supported (I don't know how to find out what version of the
>>>>>> library
>>>>>> was used for the CRAN Windows binary, but presambly it is < 4.0).
>>>>>>
>>>>>> Cheers, Mike.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  Regards,
>>>>>>> Pascal
>>>>>>>
>>>>>>> On Wed, Jun 11, 2014 at 9:10 AM, Pablo Alfaro <
>>>>>>> palfaro at motionsoft.com.uy>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, I'm having trouble opening a large netcdf file (2.45 GB) with the
>>>>>>>> following code on R 3.0.3 64 bit
>>>>>>>>
>>>>>>>> require('ncdf')
>>>>>>>> nc <- open.ncdf('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%
>>>>>>>> 3A00')
>>>>>>>> get.var.ncdf(nc, 'T2', start=c(1, 1, 1), count=c(-1, -1, -1))
>>>>>>>>
>>>>>>>> I get this error:
>>>>>>>> Assertion failed!
>>>>>>>>
>>>>>>>> Program: C:\Program Files\RStudio\bin\x64\rsession.exe
>>>>>>>> File: posixio.c, Line 417
>>>>>>>>
>>>>>>>> Expression: offset >= 0
>>>>>>>>
>>>>>>>> This application has requested the Runtime to terminate it in an
>>>>>>>>
>>>>>>> unusual
>>>>>>>
>>>>>>>> way.
>>>>>>>> Please contact the application's support team for more information.
>>>>>>>>
>>>>>>>>
>>>>>>>> In R 32 bit there are no problems.
>>>>>>>>
>>>>>>>> It is the same problem as reported here:
>>>>>>>>
>>>>>>>>  http://r.789695.n4.nabble.com/Problems-with-Raster-and-clim-
>>>>>>> pact-packages-with-large-netcdf-files-2-7G-in-x64-bit-
>>>>>>> R-td4119638.html#a4691956
>>>>>>>
>>>>>>>>
>>>>>>>> Does anyone have any clue as to why this may be?
>>>>>>>> I can upload the file on an FTP server but it'll take me a while.
>>>>>>>>
>>>>>>>> Thanks in advance!
>>>>>>>> Pablo
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ing. Pablo Alfaro Pi??eyro
>>>>>>>>
>>>>>>>> MotionSoft Consulting S.R.L.
>>>>>>>> Tel: 095 938 487
>>>>>>>>
>>>>>>>>         [[alternative HTML version deleted]]
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> R-sig-Geo mailing list
>>>>>>>> R-sig-Geo at r-project.org
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pascal Oettli
>>>>>>> Project Scientist
>>>>>>> JAMSTEC
>>>>>>> Yokohama, Japan
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> R-sig-Geo mailing list
>>>>>>> R-sig-Geo at r-project.org
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Michael Sumner
>>>>>> Software and Database Engineer
>>>>>> Australian Antarctic Division
>>>>>> Hobart, Australia
>>>>>> e-mail: mdsumner at gmail.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ing. Pablo Alfaro Pi??eyro
>>>>>
>>>>> MotionSoft Consulting S.R.L.
>>>>> Tel: 095 938 487
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Roger Bivand
>>> Department of Economics, Norwegian School of Economics,
>>> Helleveien 30, N-5045 Bergen, Norway.
>>> voice: +47 55 95 93 55; fax +47 55 95 91 00
>>> e-mail: Roger.Bivand at nhh.no
>>>
>>>
>>
>>
>> --
>> Ing. Pablo Alfaro Piñeyro
>>
>> MotionSoft Consulting S.R.L.
>> Tel: 095 938 487
>>
>
>
>
> --
> Michael Sumner
> Software and Database Engineer
> Australian Antarctic Division
> Hobart, Australia
> e-mail: mdsumner at gmail.com
>
>         [[alternative HTML version deleted]]
>
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007