> Various questions about missing values in the R ncdf package, and how
they are
> handled if the file lacks the standard "_FillValue" attribute.

Hi Andre,

It sounds like the fundamental problem is that your data files are using a
missval, but that fact is not recorded in the file's metadata as an
attribute of the variable (although it should be). The "best" solution is
to fix the problematic data files by putting in the correct missval. You
can do this with R using the ncdf package. For example, to add this
attribute to a file, you could do something like this (untested code, just
writing off the top of my head):

varname = 'Temperature'   # for example
new_missval = 9.96921e+36
ncid = open.ncdf( 'filename.nc', write=TRUE )
att.put.ncdf( ncid, varname, '_FillValue', new_missval )
close.ncdf( ncid )

Note that the more commonly accepted attribute name nowadays for a missing
value is "_FillValue", not "missing_value".

You can also do this outside of R by using the netcdf operators (nco),
http://nco.sourceforge.net/.

If fixing the files to be correct is not practical, you can always fix the
data after you read it in:

file_missval = 9.96921e+36
mvtol = abs(file_missval)*1.e-5
varname = 'Temperature'   # for example
ncid = open.ncdf( 'filename.nc' )
data = get.var.ncdf( ncid, varname )
is_missing = (abs( data - file_missval) < mvtol )  # floating comparison
data[ is_missing ] = NA

To use R to see if the missing_value of _FillValue attribute actually
exists in the file:

varname = 'Temperature'   # for example
ncid = open.ncdf( 'filename.nc' )
mv = att.get.ncdf( ncid, varname, '_FillValue" )
print(mv$hasatt)

This will be TRUE if the variable has the attribute, and FALSE otherwise.
Of course, you can repeat this for an attribute named "missing_value"
instead of "_FillValue".

See also the R documentation page for ncdf routine "set.missval.ncdf",
which allows you to change the actual missing values that are written in
the data file.

Regards,

--Dave



On Thu, Feb 6, 2014 at 7:26 AM, Dusterhus, Andre <andhus@noc.ac.uk> wrote:

> Hi all,
>
> I got currently some trouble with missvals in netCDF files:
>
> I have a netCDF file, written with an unknown program, which shows via
> ncdump no fillval or missval for its variables. Within the variables
> obviously a missval of 9.96921e+36 is used.
>
> When I read it into R via the ncdf package it shows me a missval of 1e+30,
> but the 9.96921e+36 are still in the dataset.
>
> Since my application is an automised processing of such data I have the
> following questions:
>
> Is R showing a missval, even when it is not existent in the file itself (I
> know that it uses the 1e+30 value as a default itself, when it writes
> files)?
> Could it be the case that R (ncdf) varies here from the "netCDF standard",
> which assumes 9.96921e+36 instead of 1e+30 as the standard missval (at
> least a lot of software on the market does this)?
> How is it possible via R (ncdf) to find out, whether the actual given
> missval is really given in the data?
>
> Many thanks in advance,
> André Düsterhus
>
> This message (and any attachments) is for the recipien...{{dropped:21}}


