[R-sig-Geo] Large NetCDF file problems

Thu Jun 12 05:58:27 CEST 2014

On Thu, 12 Jun 2014, Pablo Alfaro wrote:

> So you think this is not an R problem then?
> The executable failing the assertion is rsession.exe

You were told to run this outside RStudio (rsession.exe is part of RStudio 
that calls R.dll). You must do this to create a helpful reproducible 
example. The file you refer to (posixio.c) is part of NetCDF, not ncdf, 
not RNetCDF, and certainly not R.

However, there have been other examples of RStudio failing on large files 
(large spatial weights files IIRC) when the same script run directly in R 
did not fail. You need to approach this layer by layer, so removing 
RStudio is the first step.

The next is making a failing NetCDF file and its creation metadata 
available online and providing a link to it. The metadata is needed to 
know where the file is originated, as you haven't ruled out the 
possibility that the problem is in the file rather than NetCDF.

Roger

>
>
> 2014-06-11 18:14 GMT-03:00 Roger Bivand <Roger.Bivand at nhh.no>:
>
>> I suggest that you put together a complete reproducible example with
>> access to the file on a server, and send this to the maintainers of ncdf
>> and RNetCDF. This looks either like an issue in compiler flags, or an issue
>> in the code of the external library used.
>>
>> Roger
>>
>>
>> On Wed, 11 Jun 2014, Pablo Alfaro wrote:
>>
>>  Hmm, just tried using RNetCDF (v 1.6.2-2) and it has the same problem.
>>>
>>> When using
>>>
>>> require('RNetCDF')
>>> nc <- open.nc('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%3A00')
>>> var.get.nc(nc, "T2", start=c(1, 1, 1), count=c(NA, NA, NA))
>>>
>>> I get the same problem on 64bit R, no problem on 32bits.
>>>
>>> Seems like a problem with the version of posixio.c used in 64 bit R. Just
>>> found a version of posixio.c on the web (
>>> http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/util/sorc/
>>> ncdump.fd/libsrc/posixio.c)
>>> which has the assertion on line 417.
>>>
>>> assert(offset >= 0);
>>>
>>> offset is a variable of type off_t which is defined in <sys/types.h> and
>>> apparently requires that you enable Large File Support on the compiler
>>> command line to be a 64 bit integer.
>>> Can it be that the 32 bit version of R is using Large File Support but the
>>> 64 bit version is not?
>>>
>>> Best
>>> Pablo
>>>
>>>
>>>
>>>
>>> 2014-06-11 15:07 GMT-03:00 Pablo Alfaro <palfaro at motionsoft.com.uy>:
>>>
>>>  Hi Michael, Pascal, thank you both for your replies.
>>>>
>>>> Yeah, Sorry, I'm using R 3.0.3 64 (or 32 for the working version) bit, on
>>>> Windows 7 x64 and the NetCDF library is ncdf (not ncdf4) version 1.6.6,
>>>> the
>>>> last available version.
>>>>
>>>> I need it to run on Windows since the servers I run the code on are
>>>> Windows.
>>>>
>>>> I tried downloading and installing the package you mention Michael but
>>>> when loading it, it says the x64 architecture is not supported (the
>>>> author
>>>> mentions it in the site as well). The thing is that ncdf works on x86 so
>>>> it
>>>> leaves me with the same options.
>>>>
>>>> RNetCDF seems to use NetCDF v3, but maybe it doesn't have this problem. I
>>>> wil try that and see if it helps.
>>>>
>>>> The assertion on the error complains an offset which isn't >= 0. If I
>>>> were
>>>> to take a guess, that offset is a 32 bit signed integer and that's why it
>>>> breaks only with large files (bigger than 2^31 - 1 bytes). But I can't
>>>> tell
>>>> for sure. Also it's weird that it works on 32 bits.
>>>>
>>>> Best
>>>> Pablo
>>>>
>>>>
>>>> 2014-06-10 23:13 GMT-03:00 Michael Sumner <mdsumner at gmail.com>:
>>>>
>>>>
>>>>
>>>>>
>>>>> On Wed, Jun 11, 2014 at 11:49 AM, Pascal Oettli <kridox at ymail.com>
>>>>> wrote:
>>>>>
>>>>>  Hello,
>>>>>>
>>>>>> Do you get the same error if you run your code out of Rstudio?
>>>>>>
>>>>>> However, another approach would be to open your NetCDF file with
>>>>>> 'brick' function from the raster package.
>>>>>>
>>>>>>
>>>>>
>>>>> I don't think either of these will help, raster simply runs ncdf (or
>>>>> preferably ncdf4) to do the file read. You might find it easier to move
>>>>> to
>>>>> ncdf4 if you can, but potentially the stuff I write below is just an
>>>>> unwelcome aside. I have seen a similar error in a file I've tried to
>>>>> use,
>>>>> so I'll have another look at that and see if I can isolate where the
>>>>> problem lies. At minimum we really need to know the version of the
>>>>> NetCDF
>>>>> library in use and perhaps details of how it was compiled so at minimum
>>>>> you
>>>>> should provide the ncdf package version as well as R's.
>>>>>
>>>>> It's possible to install ncdf4 on Windows, but it's not supported (no
>>>>> binary .zip on CRAN) but you can do it yourself (again not supported,
>>>>> and
>>>>> the archive here is not updated):
>>>>> http://cirrus.ucsd.edu/~pierce/ncdf/
>>>>>
>>>>> It's much more generally available to do this on Linux, though there's
>>>>> always bumps in the road.
>>>>>
>>>>> RNetCDF is actively updated on CRAN and is another option to see if the
>>>>> read is supported (I don't know how to find out what version of the
>>>>> library
>>>>> was used for the CRAN Windows binary, but presambly it is < 4.0).
>>>>>
>>>>> Cheers, Mike.
>>>>>
>>>>>
>>>>>
>>>>>  Regards,
>>>>>> Pascal
>>>>>>
>>>>>> On Wed, Jun 11, 2014 at 9:10 AM, Pablo Alfaro <
>>>>>> palfaro at motionsoft.com.uy>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, I'm having trouble opening a large netcdf file (2.45 GB) with the
>>>>>>> following code on R 3.0.3 64 bit
>>>>>>>
>>>>>>> require('ncdf')
>>>>>>> nc <- open.ncdf('C:/testsMCH/test/wrfout_d02_2014-06-09_00%3A00%
>>>>>>> 3A00')
>>>>>>> get.var.ncdf(nc, 'T2', start=c(1, 1, 1), count=c(-1, -1, -1))
>>>>>>>
>>>>>>> I get this error:
>>>>>>> Assertion failed!
>>>>>>>
>>>>>>> Program: C:\Program Files\RStudio\bin\x64\rsession.exe
>>>>>>> File: posixio.c, Line 417
>>>>>>>
>>>>>>> Expression: offset >= 0
>>>>>>>
>>>>>>> This application has requested the Runtime to terminate it in an
>>>>>>>
>>>>>> unusual
>>>>>>
>>>>>>> way.
>>>>>>> Please contact the application's support team for more information.
>>>>>>>
>>>>>>>
>>>>>>> In R 32 bit there are no problems.
>>>>>>>
>>>>>>> It is the same problem as reported here:
>>>>>>>
>>>>>>>  http://r.789695.n4.nabble.com/Problems-with-Raster-and-clim-
>>>>>> pact-packages-with-large-netcdf-files-2-7G-in-x64-bit-
>>>>>> R-td4119638.html#a4691956
>>>>>>
>>>>>>>
>>>>>>> Does anyone have any clue as to why this may be?
>>>>>>> I can upload the file on an FTP server but it'll take me a while.
>>>>>>>
>>>>>>> Thanks in advance!
>>>>>>> Pablo
>>>>>>>
>>>>>>> --
>>>>>>> Ing. Pablo Alfaro Pi??eyro
>>>>>>>
>>>>>>> MotionSoft Consulting S.R.L.
>>>>>>> Tel: 095 938 487
>>>>>>>
>>>>>>>         [[alternative HTML version deleted]]
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> R-sig-Geo mailing list
>>>>>>> R-sig-Geo at r-project.org
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Pascal Oettli
>>>>>> Project Scientist
>>>>>> JAMSTEC
>>>>>> Yokohama, Japan
>>>>>>
>>>>>> _______________________________________________
>>>>>> R-sig-Geo mailing list
>>>>>> R-sig-Geo at r-project.org
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Michael Sumner
>>>>> Software and Database Engineer
>>>>> Australian Antarctic Division
>>>>> Hobart, Australia
>>>>> e-mail: mdsumner at gmail.com
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Ing. Pablo Alfaro Pi??eyro
>>>>
>>>> MotionSoft Consulting S.R.L.
>>>> Tel: 095 938 487
>>>>
>>>>
>>>
>>>
>>>
>>>
>> --
>> Roger Bivand
>> Department of Economics, Norwegian School of Economics,
>> Helleveien 30, N-5045 Bergen, Norway.
>> voice: +47 55 95 93 55; fax +47 55 95 91 00
>> e-mail: Roger.Bivand at nhh.no
>>
>>
>
>
>

-- 
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 91 00
e-mail: Roger.Bivand at nhh.no