[R-sig-Geo] write/readOGR issue with date / time strings

Roger Bivand Roger.Bivand at nhh.no
Wed Nov 16 14:33:56 CET 2011 

On Tue, 15 Nov 2011, Roger Bivand wrote:

> On Tue, 15 Nov 2011, Corrie Curtice wrote:
>
>> Hi Roger,
>> 
>> Thanks for your patience. This one is confusing to me as I've done exactly
>> (well, I think exactly) the same steps in the past and it has worked fine.
>> The only difference is (a) I'm using Rstudio which made me (b) update my
>> rgdal library version.  I've always converted date/time fields using
>> as.character or format().  Answers to your questions embedded below...
>> 
>
> Nothing has changed that should cause this misbehaviour. Can you save() the 
> spatial points data frame and put it on a link for me to pick up to try to 
> reproduce your problem on a different platform? The GDAL release isn't the 
> issue, I think. Did you say how the data was read into R?
>

The issue was that in columns/fields with names over 10 characters, the 
values were not written (in OGR for the ESRI Shapefile driver). The 
work-around - use short field names of 10 characters or fewer. A fresh 
rgdal with a fix which arbitrarily abbreviates names has been submitted to 
CRAN.

Roger

> Roger
>
>> 
>> On 11/12/11 3:17 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:
>> 
>> 
>>> On Fri, 11 Nov 2011, Corrie Curtice wrote:
>>> 
>>>> 
>>>>> sessionInfo()
>>>> R version 2.13.1 (2011-07-08)
>>>> Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
>>>> 
>>>> locale:
>>>> [1] C/en_US.UTF-8/C/C/C/C
>>>> 
>>>> attached base packages:
>>>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>>> 
>>>> other attached packages:
>>>> [1] rgdal_0.7-1 sp_0.9-84
>>> 
>>> OK, you have the most recent release.
>>> 
>>>> 
>>>> loaded via a namespace (and not attached):
>>>> [1] grid_2.13.1     lattice_0.19-30 tools_2.13.1
>>>> 
>>>> I installed rgdal and PROJ4 as follows:
>>>> 
>>>> setRepositories(ind=1:2)
>>>> install.packages("rgdal")
>>>> install.packages("PROJ.4")
>>>> 
>>> 
>>> Quoting the loading messages would help, to confirm the version of the
>>> underlying GDAL binary. The PROJ.4 package is not required or used by
>>> rgdal. My rgdal version on Fedora reports:
>>> 
>>>> library(rgdal)
>>> Loading required package: sp
>>> Geospatial Data Abstraction Library extensions to R successfully loaded
>>> Loaded GDAL runtime: GDAL 1.8.1, released 2011/06/04
>>> Path to GDAL shared files: /usr/local/share/gdal
>>> Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470]
>>> Path to PROJ.4 shared files: (autodetected)
>>> 
>> 
>> Here's the output from loading rgdal:
>> 
>>> library(rgdal)
>> Geospatial Data Abstraction Library extensions to R successfully loaded
>> Loaded GDAL runtime: GDAL 1.8.0, released 2011/01/12
>> Path to GDAL shared files:
>> /Users/corriecurtice/Library/R/2.13/library/rgdal/gdal
>> Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470]
>> Path to PROJ.4 shared files:
>> /Users/corriecurtice/Library/R/2.13/library/rgdal/proj
>>> 
>> 
>> So there's a slightly newer GDAL release (1.8.1).  I'm in the process of
>> trying to get that installed but not having a lot of luck.  I downloaded
>> and installed gdal v1.8.1-1 leopard (and dependencies). I assume I need to
>> re-compile the rgdal package so it sees this new version but I get an
>> error when trying to do that:
>> 
>> corrie-curtices-macbook-pro:Downloads corriecurtice$ !R
>> R CMD INSTALL
>> --configure-args='--with-gdal-config=/Library/Frameworks/GDAL.framework/Pro
>> grams/gdal-config
>> --with-proj-include=/Library/Frameworks/PROJ.framework/Headers
>> --with-proj-lib=/Library/Frameworks/PROJ.framework/unix/lib'
>> rgdal_0.7-4.tar.gz
>> * installing to library ?/Users/corriecurtice/Library/R/2.13/library¹
>> * installing *source* package ?rgdal¹ ...
>> gdal-config: /Library/Frameworks/GDAL.framework/Programs/gdal-config
>> checking for gcc... gcc-4.2 -arch i386 -std=gnu99
>> checking whether the C compiler works... no
>> configure: error: in
>> `/private/var/folders/jD/jDD8BZCHHLC-VOQmq7vTGk+++TI/-Tmp-/RtmpTlKhSZ/R.INS
>> TALL253fc47f/rgdal':
>> configure: error: C compiler cannot create executables
>> See `config.log' for more details
>> ERROR: configuration failed for package ?rgdal¹
>> * removing ?/Users/corriecurtice/Library/R/2.13/library/rgdal¹
>> * restoring previous ?/Users/corriecurtice/Library/R/2.13/library/rgdal¹
>> 
>> 
>> I can't find the config.log file, the tmp directory referenced is empty
>> and it's not in my cwd.
>> 
>>>> 
>>>> 
>>> 
>>> I do not see the problem in the R-forge development version, and cannot
>>> reproduce the problem. This now works in (unreleased) 0.7-4:
>>> 
>>> library(rgdal)
>>> dsn <- system.file("vectors", package = "rgdal")[1]
>>>   kiritimati_primary_roads <- readOGR(dsn=dsn,
>>> layer="kiritimati_primary_roads")
>>> summary(kiritimati_primary_roads)
>>> kiritimati_primary_roads$dt <- as.POSIXct("2011-11-11 11:11:11")+(60*0:34)
>>> summary(kiritimati_primary_roads)
>>> td <- tempdir()
>>> writeOGR(kiritimati_primary_roads, td, "kiri1", driver="ESRI Shapefile")
>>> kiri1 <- readOGR(td, "kiri1")
>>> summary(kiri1)
>>> kiri1$dt <- as.POSIXct(as.character(kiri1$dt))
>>> summary(kiri1)
>>> 
>>> This failed in 0.7-1 because the POSIXct column is not recognized as also
>>> POSIXt; this is corrected in 0.7-4. However, this was not the original
>>> problem. Had you reported the output of:
>>> 
>>> sapply(spdfUTM at data, class)
>>> sapply(spdfUTM at data, storage.mode)
>>> 
>>> we would have known how krillGMTtime was stored - it does not seem to
>>> have
>>> been detected as a column requiring conversion to character string, or as
>>> a column that should trigger an error. In general, as Alex wrote, all
>>> date
>>> time columns should be converted to character and converted back on
>>> reading back from the shapefile. OGR drivers treat field types
>>> differently, so writeOGR has to take a conservative view of possibilities.
>> 
>> Here's the output from the sapply:
>> 
>> 
>>> sapply(spdfUTM at data, class)
>> krillGMTtime          Lat          Lon
>>    "factor"     "factor"     "factor"
>>> sapply(spdfUTM at data, storage.mode)
>> krillGMTtime          Lat          Lon
>>   "integer"    "integer"    "integer"
>> 
>> 
>> To answer Don's question, yes I have bad LAT/LON rows when I first read
>> them in but they are all 999. I discard them before making my spatial
>> points data frame, which does in fact use those two columns for the
>> coordinates. I have another similar dataset where the date/time field
>> storage.mode is reported as "character" and not "integer", and has the
>> same issue.
>> 
>> 
>> Thanks for the help.
>> 
>> Corrie
>> 
>> 
>>> Hope this clarifies,
>>> 
>>> Roger
>>> 
>>>> Thanks,
>>>> 
>>>> Corrie
>>>> 
>>>> 
>>>> 
>>>> On 11/11/11 5:31 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:
>>>> 
>>>>> On Fri, 11 Nov 2011, Corrie Curtice wrote:
>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> Apologies if this has already been addressed somewhere, I did a brief
>>>>>> search
>>>>>> of archives but didn't find quite this issue.
>>>>> 
>>>>> Which versions of what are you using? You should provide the output of
>>>>> sessionInfo(), the messages given when rgdal loads, and details of how
>>>>> you
>>>>> installed rgdal. In addition, a use case would be helpful, but only
>>>>> when
>>>>> we know what version you are using. In current rgdal, data frame
>>>>> columns
>>>>> of class POSIXt are converted to character strings before writing with
>>>>> writeOGR().
>>>>> 
>>>>> Roger
>>>>> 
>>>>>> 
>>>>>> I'm writing out an ESRI shapefile.  My spdf has a date/time field.
>>>>>> Looks
>>>>>> like this:
>>>>>> 
>>>>>>> head(spdfUTM at data)
>>>>>>           krillGMTtime          Lat          Lon
>>>>>> 1   2010-05-12 12:34:21 -64.67655969 -62.15040195
>>>>>> 21  2010-05-12 12:35:12  -64.6771229  -62.1519511
>>>>>> 38  2010-05-12 12:36:02 -64.67775863 -62.15340614
>>>>>> 57  2010-05-12 12:36:53 -64.67838269 -62.15494829
>>>>>> 78  2010-05-12 12:37:43 -64.67901497 -62.15647203
>>>>>> 100 2010-05-12 12:38:37 -64.67973667 -62.15773224
>>>>>> 
>>>>>> All three fields are reported to be "factors" by str.  writeOGR
>>>>>> appears
>>>>>> happy:
>>>>>> 
>>>>>>> writeOGR(spdfUTM,dd,layer="krillPoints-UTM",driver="ESRI
>>>>>> Shapefile",verbose=TRUE,overwrite=TRUE)
>>>>>> $object_type
>>>>>> [1] "SpatialPointsDataFrame"
>>>>>> $output_dsn
>>>>>> [1] "/users/corriecurtice/documents/Data_2010/Shapefiles/"
>>>>>> $output_layer
>>>>>> [1] "krillPoints-UTM"
>>>>>> $output_diver
>>>>>> [1] "ESRI Shapefile"
>>>>>> $output_n
>>>>>> [1] 11179
>>>>>> $output_nfields
>>>>>> [1] 3
>>>>>> $output_fields
>>>>>> [1] "krillGMTtime" "Lat"          "Lon"
>>>>>> $output_fclasses
>>>>>> [1] 4 4 4
>>>>>> $dataset_options
>>>>>> NULL
>>>>>> $layer_options
>>>>>> NULL
>>>>>> Warning message:
>>>>>> In writeOGR(spdfUTM, dd, layer = "krillPoints-UTM", driver = "ESRI
>>>>>> Shapefile",  :
>>>>>>  existing layer removed
>>>>>> 
>>>>>> When I read it right back in again, the date/time field is NAs.  This
>>>>>> is
>>>>>> also true if I load the shapefile into ArcMap.
>>>>>> 
>>>>>>> foo <- readOGR(dd,layer="krillPoints-UTM",verbose=TRUE)
>>>>>> OGR data source with driver: ESRI Shapefile
>>>>>> Source: "/users/corriecurtice/documents/Data_2010/Shapefiles/", layer:
>>>>>> "krillPoints-UTM"
>>>>>> with 11179 features and 3 fields
>>>>>> Feature type: wkbPoint with 2 dimensions
>>>>>> 
>>>>>>> head(foo at data)
>>>>>>  krillGMTti          Lat          Lon
>>>>>> 1       <NA> -64.67655969 -62.15040195
>>>>>> 2       <NA>  -64.6771229  -62.1519511
>>>>>> 3       <NA> -64.67775863 -62.15340614
>>>>>> 4       <NA> -64.67838269 -62.15494829
>>>>>> 5       <NA> -64.67901497 -62.15647203
>>>>>> 6       <NA> -64.67973667 -62.15773224
>>>>>> 
>>>>>> Thoughts? Am I missing something obvious?
>>>>>> 
>>>>>> I can break it up into separate date and time fields, but since the
>>>>>> field is
>>>>>> simply a factor anyways I'm not sure how that would help.
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Corrie
>>>>>> 
>>>>>> ---
>>>>>> Corrie Curtice
>>>>>> Research Analyst
>>>>>> Marine Geospatial Ecology Lab
>>>>>> Nicholas School of the Environment, Duke University
>>>>>> http://mgel.env.duke.edu
>>>>>> em: corrie.curtice at duke.edu
>>>>>> ph: 252-504-7538
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>
>>>>>>     [[alternative HTML version deleted]]
>>>>>> 
>>>>>> _______________________________________________
>>>>>> R-sig-Geo mailing list
>>>>>> R-sig-Geo at r-project.org
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>> 
>>>>> 
>>>>> --
>>>>> Roger Bivand
>>>>> Department of Economics, NHH Norwegian School of Economics,
>>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>>> voice: +47 55 95 93 55; fax +47 55 95 95 43
>>>>> e-mail: Roger.Bivand at nhh.no
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> R-sig-Geo mailing list
>>>> R-sig-Geo at r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>> 
>>> 
>>> --
>>> Roger Bivand
>>> Department of Economics, NHH Norwegian School of Economics,
>>> Helleveien 30, N-5045 Bergen, Norway.
>>> voice: +47 55 95 93 55; fax +47 55 95 95 43
>>> e-mail: Roger.Bivand at nhh.no
>> 
>> 
>> 
>
>

-- 
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

More information about the R-sig-Geo mailing list