[R-sig-Geo] Large NetCDF file problems

Wed Jun 11 23:14:41 CEST 2014

I suggest that you put together a complete reproducible example with 
access to the file on a server, and send this to the maintainers of ncdf 
and RNetCDF. This looks either like an issue in compiler flags, or an 
issue in the code of the external library used.

Roger

On Wed, 11 Jun 2014, Pablo Alfaro wrote:

> Hmm, just tried using RNetCDF (v 1.6.2-2) and it has the same problem.
>
> When using
>
> require('RNetCDF')
> nc <- open.nc('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%3A00')
> var.get.nc(nc, "T2", start=c(1, 1, 1), count=c(NA, NA, NA))
>
> I get the same problem on 64bit R, no problem on 32bits.
>
> Seems like a problem with the version of posixio.c used in 64 bit R. Just
> found a version of posixio.c on the web (
> http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/util/sorc/ncdump.fd/libsrc/posixio.c)
> which has the assertion on line 417.
>
> assert(offset >= 0);
>
> offset is a variable of type off_t which is defined in <sys/types.h> and
> apparently requires that you enable Large File Support on the compiler
> command line to be a 64 bit integer.
> Can it be that the 32 bit version of R is using Large File Support but the
> 64 bit version is not?
>
> Best
> Pablo
>
>
>
>
> 2014-06-11 15:07 GMT-03:00 Pablo Alfaro <palfaro at motionsoft.com.uy>:
>
>> Hi Michael, Pascal, thank you both for your replies.
>>
>> Yeah, Sorry, I'm using R 3.0.3 64 (or 32 for the working version) bit, on
>> Windows 7 x64 and the NetCDF library is ncdf (not ncdf4) version 1.6.6, the
>> last available version.
>>
>> I need it to run on Windows since the servers I run the code on are
>> Windows.
>>
>> I tried downloading and installing the package you mention Michael but
>> when loading it, it says the x64 architecture is not supported (the author
>> mentions it in the site as well). The thing is that ncdf works on x86 so it
>> leaves me with the same options.
>>
>> RNetCDF seems to use NetCDF v3, but maybe it doesn't have this problem. I
>> wil try that and see if it helps.
>>
>> The assertion on the error complains an offset which isn't >= 0. If I were
>> to take a guess, that offset is a 32 bit signed integer and that's why it
>> breaks only with large files (bigger than 2^31 - 1 bytes). But I can't tell
>> for sure. Also it's weird that it works on 32 bits.
>>
>> Best
>> Pablo
>>
>>
>> 2014-06-10 23:13 GMT-03:00 Michael Sumner <mdsumner at gmail.com>:
>>
>>
>>>
>>>
>>> On Wed, Jun 11, 2014 at 11:49 AM, Pascal Oettli <kridox at ymail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Do you get the same error if you run your code out of Rstudio?
>>>>
>>>> However, another approach would be to open your NetCDF file with
>>>> 'brick' function from the raster package.
>>>>
>>>
>>>
>>> I don't think either of these will help, raster simply runs ncdf (or
>>> preferably ncdf4) to do the file read. You might find it easier to move to
>>> ncdf4 if you can, but potentially the stuff I write below is just an
>>> unwelcome aside. I have seen a similar error in a file I've tried to use,
>>> so I'll have another look at that and see if I can isolate where the
>>> problem lies. At minimum we really need to know the version of the NetCDF
>>> library in use and perhaps details of how it was compiled so at minimum you
>>> should provide the ncdf package version as well as R's.
>>>
>>> It's possible to install ncdf4 on Windows, but it's not supported (no
>>> binary .zip on CRAN) but you can do it yourself (again not supported, and
>>> the archive here is not updated):
>>> http://cirrus.ucsd.edu/~pierce/ncdf/
>>>
>>> It's much more generally available to do this on Linux, though there's
>>> always bumps in the road.
>>>
>>> RNetCDF is actively updated on CRAN and is another option to see if the
>>> read is supported (I don't know how to find out what version of the library
>>> was used for the CRAN Windows binary, but presambly it is < 4.0).
>>>
>>> Cheers, Mike.
>>>
>>>
>>>
>>>> Regards,
>>>> Pascal
>>>>
>>>> On Wed, Jun 11, 2014 at 9:10 AM, Pablo Alfaro <palfaro at motionsoft.com.uy>
>>>> wrote:
>>>>> Hi, I'm having trouble opening a large netcdf file (2.45 GB) with the
>>>>> following code on R 3.0.3 64 bit
>>>>>
>>>>> require('ncdf')
>>>>> nc <- open.ncdf('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%3A00')
>>>>> get.var.ncdf(nc, 'T2', start=c(1, 1, 1), count=c(-1, -1, -1))
>>>>>
>>>>> I get this error:
>>>>> Assertion failed!
>>>>>
>>>>> Program: C:\Program Files\RStudio\bin\x64\rsession.exe
>>>>> File: posixio.c, Line 417
>>>>>
>>>>> Expression: offset >= 0
>>>>>
>>>>> This application has requested the Runtime to terminate it in an
>>>> unusual
>>>>> way.
>>>>> Please contact the application's support team for more information.
>>>>>
>>>>>
>>>>> In R 32 bit there are no problems.
>>>>>
>>>>> It is the same problem as reported here:
>>>>>
>>>> http://r.789695.n4.nabble.com/Problems-with-Raster-and-clim-pact-packages-with-large-netcdf-files-2-7G-in-x64-bit-R-td4119638.html#a4691956
>>>>>
>>>>> Does anyone have any clue as to why this may be?
>>>>> I can upload the file on an FTP server but it'll take me a while.
>>>>>
>>>>> Thanks in advance!
>>>>> Pablo
>>>>>
>>>>> --
>>>>> Ing. Pablo Alfaro Pi??eyro
>>>>> MotionSoft Consulting S.R.L.
>>>>> Tel: 095 938 487
>>>>>
>>>>>         [[alternative HTML version deleted]]
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-Geo mailing list
>>>>> R-sig-Geo at r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pascal Oettli
>>>> Project Scientist
>>>> JAMSTEC
>>>> Yokohama, Japan
>>>>
>>>> _______________________________________________
>>>> R-sig-Geo mailing list
>>>> R-sig-Geo at r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>
>>>
>>>
>>>
>>> --
>>> Michael Sumner
>>> Software and Database Engineer
>>> Australian Antarctic Division
>>> Hobart, Australia
>>> e-mail: mdsumner at gmail.com
>>>
>>
>>
>>
>> --
>> Ing. Pablo Alfaro Pi??eyro
>> MotionSoft Consulting S.R.L.
>> Tel: 095 938 487
>>
>
>
>
>

-- 
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 91 00
e-mail: Roger.Bivand at nhh.no