[R-sig-Geo] Problem with raster package and netcdf file

[Michael Sumner]

FWIW, this should now work out of the box with the release of
ncdf4_1.10, and previous updates in raster long ago.

## (as attempted by OP)
bat <- raster(file="gebco_08_-82_37_63_80",varname="z")

I'll check it when I have a chance to build the package, but any other
verification would be nice.

Cheers, Mike.

On Thu, Mar 14, 2013 at 10:11 PM, Michael Sumner <mdsumner at gmail.com> wrote:
> This should now work if you upgrade to raster_2.1-12.
>
> It will still use as much memory as is contained in the file, just to open
> it deep inside the raster() handling of the connection - but it's not
> permanent and if you can get past this step, try running gc() and it should
> settle down (I'm on Windows with ncdf_1.6.6):
>
> library(raster) ## 2.1-12 or greater
> bat <- raster(file="gebco_08_-82_37_63_80")
>
> gc()
>
> (I've only tested on gebco_08.nc, the original large file itself). Do not
> set varname = "z", it should choose that anyway.
>
> I'm hoping that a change in ncdf4 will be published to avoid the memory
> consumption, but I need to put the case for change in ncdf, ncdf4 and raster
> which potentially involves 3 separate package maintainers. They have each
> been very patient with me so far.
>
> If you want to shortcut to a solution, just use gdal_translate from OSGeo4W,
> and write to GeoTIFF with (optionally) lossless compression and tiling:
>
> gdal_translate gebco_08_-82_37_63_80 gebco_08_-82_37_63_80.tif -co
> COMPRESS=LZW -co TILED=YES
>
> BTW, It's a strange coincidence that this was asked this week just when I
> was investigating why it did not already work!  The GDAL workaround is good,
> but raster/ncdf/4 is a very nice general fix.
>
> And, for what it is worth - GDAL has long supported this NetCDF variant, so
> a build of rgdal with the GMT driver  should work well too with the extra
> caveats of subsetting with readGDAL() or via a GDALMajorObject. (CRAN's
> Windows rgdal does not have GMT)
>
> Cheers, Mike.
>
>
>
>
>
> On Sat, Mar 9, 2013 at 3:24 PM, Michael Sumner <mdsumner at gmail.com> wrote:
>>
>> The raster package isn't able to read the data from this "GMT-style"
>> NetCDF, since the structure of the variable is one-dimensional internally
>> and you are supposed to deal with that yourself. I would stick with the
>> ncdf4 package to read it, you'll have to scan into the "z" variable manually
>> to reconstruct it as a matrix, or subset it.
>>
>> Note that you should have a large object size with "d", since the nc_open
>> function returns an index vector for each dimension (so one of them is
>> length 89784000, you must have sufficient memory to deal with that - I would
>> close/delete the "d" object once you read the "z", "dimension" and other
>> values).
>>
>> I have code to read from these files, but I can't access it today - it's
>> not that difficult though, you can probably do something like matrix(z,
>> dimension[1], dimension[2]) but you'll have to fiddle to get the orientation
>> right.
>>
>>
>>
>>
>> On Fri, Mar 8, 2013 at 9:18 PM, Julian Burgos <julian at hafro.is> wrote:
>>>
>>> Dear list,
>>>
>>> I am having some troubles opening accessing a raster in the netCDF
>>> format.
>>>  The file is bathimetric data from GEBCO:
>>> http://www.gebco.net/data_and_products/gridded_bathymetry_data/
>>>
>>> If I use the ncdf4 package to open and access the data, I have no issues:
>>>
>>> library(ncdf4)
>>> > d <- nc_open("gebco_08_-82_37_63_80.nc")
>>> > z <- as.vector(ncvar_get(d,varid="z"))
>>> > head(z)
>>> [1] 19 40 61 83 106 128
>>>
>>>
>>> But if I try to use the raster package to access the file and create a
>>> raster object, I get the following error:
>>>
>>> bat <- raster(file="gebco_08_-82_37_63_80",varname="z")
>>> Error in .varName(nc, varname, warn = warn) :
>>>   varname: z does not exist in the file. Select one from:
>>>
>>> ... and no varnames are listed.
>>>
>>>
>>> If I do not define a variable name and let the raster package choose it,
>>> I
>>> get this:
>>>
>>> > bat <- raster(file="gebco_08_-82_37_63_80",varname="z")
>>> Error in .varName(nc, varname, warn = warn) :
>>>
>>> The description of the netCDF file is as following:
>>>
>>> > d
>>> [1] "File gebco_08_-82_37_63_80.nc (NC_FORMAT_CLASSIC):"
>>> [1] ""
>>> [1] "     6 variables:"
>>> [1] "        double x_range[side]   "
>>> [1] "            units: user_x_unit"
>>> [1] "        double y_range[side]   "
>>> [1] "            units: user_y_unit"
>>> [1] "        short z_range[side]   "
>>> [1] "            units: user_z_unit"
>>> [1] "        double spacing[side]   "
>>> [1] "        int dimension[side]   "
>>> [1] "        short z[xysize]   "
>>> [1] "            scale_factor: 1"
>>> [1] "            add_offset: 0"
>>> [1] "            node_offset: 1"
>>> [1] ""
>>> [1] "     2 dimensions:"
>>> [1] "        side  Size:2"
>>> [1] "        xysize  Size:89784000"
>>> [1] ""
>>> [1] "    2 global attributes:"
>>> [1] "        title: GEBCO_08 Grid"
>>> [1] "        source: 20100927"
>>>
>>> I am using R 15.2 on Linux (Fedora), raster 2.1-8, and ncdf4 version
>>> ncdf4_1.6.1_20120411.
>>> Any help will be welcomed!
>>>
>>> Julian
>>>
>>> --
>>> Julian Mariano Burgos, PhD
>>> Hafrannsóknastofnunin/Marine Research Institute
>>> Skúlagata 4, 121 Reykjavík, Iceland
>>> Sími/Telephone : +354-5752037
>>> Bréfsími/Telefax:  +354-5752001
>>> Netfang/Email: julian at hafro.is
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>>
>>
>>
>> --
>> Michael Sumner
>> Hobart, Australia
>> e-mail: mdsumner at gmail.com
>
>
>
>
> --
> Michael Sumner
> Hobart, Australia
> e-mail: mdsumner at gmail.com



-- 
Michael Sumner
Hobart, Australia
e-mail: mdsumner at gmail.com