[R-sig-Geo] write/readOGR issue with date / time strings

Roger Bivand Roger.Bivand at nhh.no
Tue Nov 15 19:37:30 CET 2011 

On Tue, 15 Nov 2011, Corrie Curtice wrote:

> Hi Roger,
>
> Thanks for your patience. This one is confusing to me as I've done exactly
> (well, I think exactly) the same steps in the past and it has worked fine.
> The only difference is (a) I'm using Rstudio which made me (b) update my
> rgdal library version.  I've always converted date/time fields using
> as.character or format().  Answers to your questions embedded below...
>

Nothing has changed that should cause this misbehaviour. Can you save() 
the spatial points data frame and put it on a link for me to pick up to 
try to reproduce your problem on a different platform? The GDAL release 
isn't the issue, I think. Did you say how the data was read into R?

Roger

>
> On 11/12/11 3:17 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:
>
>
>> On Fri, 11 Nov 2011, Corrie Curtice wrote:
>>
>>>
>>>> sessionInfo()
>>> R version 2.13.1 (2011-07-08)
>>> Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
>>>
>>> locale:
>>> [1] C/en_US.UTF-8/C/C/C/C
>>>
>>> attached base packages:
>>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>>
>>> other attached packages:
>>> [1] rgdal_0.7-1 sp_0.9-84
>>
>> OK, you have the most recent release.
>>
>>>
>>> loaded via a namespace (and not attached):
>>> [1] grid_2.13.1     lattice_0.19-30 tools_2.13.1
>>>
>>> I installed rgdal and PROJ4 as follows:
>>>
>>> setRepositories(ind=1:2)
>>> install.packages("rgdal")
>>> install.packages("PROJ.4")
>>>
>>
>> Quoting the loading messages would help, to confirm the version of the
>> underlying GDAL binary. The PROJ.4 package is not required or used by
>> rgdal. My rgdal version on Fedora reports:
>>
>>> library(rgdal)
>> Loading required package: sp
>> Geospatial Data Abstraction Library extensions to R successfully loaded
>> Loaded GDAL runtime: GDAL 1.8.1, released 2011/06/04
>> Path to GDAL shared files: /usr/local/share/gdal
>> Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470]
>> Path to PROJ.4 shared files: (autodetected)
>>
>
> Here's the output from loading rgdal:
>
>> library(rgdal)
> Geospatial Data Abstraction Library extensions to R successfully loaded
> Loaded GDAL runtime: GDAL 1.8.0, released 2011/01/12
> Path to GDAL shared files:
> /Users/corriecurtice/Library/R/2.13/library/rgdal/gdal
> Loaded PROJ.4 runtime: Rel. 4.7.1, 23 September 2009, [PJ_VERSION: 470]
> Path to PROJ.4 shared files:
> /Users/corriecurtice/Library/R/2.13/library/rgdal/proj
>>
>
> So there's a slightly newer GDAL release (1.8.1).  I'm in the process of
> trying to get that installed but not having a lot of luck.  I downloaded
> and installed gdal v1.8.1-1 leopard (and dependencies). I assume I need to
> re-compile the rgdal package so it sees this new version but I get an
> error when trying to do that:
>
> corrie-curtices-macbook-pro:Downloads corriecurtice$ !R
> R CMD INSTALL
> --configure-args='--with-gdal-config=/Library/Frameworks/GDAL.framework/Pro
> grams/gdal-config
> --with-proj-include=/Library/Frameworks/PROJ.framework/Headers
> --with-proj-lib=/Library/Frameworks/PROJ.framework/unix/lib'
> rgdal_0.7-4.tar.gz
> * installing to library ?/Users/corriecurtice/Library/R/2.13/library¹
> * installing *source* package ?rgdal¹ ...
> gdal-config: /Library/Frameworks/GDAL.framework/Programs/gdal-config
> checking for gcc... gcc-4.2 -arch i386 -std=gnu99
> checking whether the C compiler works... no
> configure: error: in
> `/private/var/folders/jD/jDD8BZCHHLC-VOQmq7vTGk+++TI/-Tmp-/RtmpTlKhSZ/R.INS
> TALL253fc47f/rgdal':
> configure: error: C compiler cannot create executables
> See `config.log' for more details
> ERROR: configuration failed for package ?rgdal¹
> * removing ?/Users/corriecurtice/Library/R/2.13/library/rgdal¹
> * restoring previous ?/Users/corriecurtice/Library/R/2.13/library/rgdal¹
>
>
> I can't find the config.log file, the tmp directory referenced is empty
> and it's not in my cwd.
>
>>>
>>>
>>
>> I do not see the problem in the R-forge development version, and cannot
>> reproduce the problem. This now works in (unreleased) 0.7-4:
>>
>> library(rgdal)
>> dsn <- system.file("vectors", package = "rgdal")[1]
>>   kiritimati_primary_roads <- readOGR(dsn=dsn,
>> layer="kiritimati_primary_roads")
>> summary(kiritimati_primary_roads)
>> kiritimati_primary_roads$dt <- as.POSIXct("2011-11-11 11:11:11")+(60*0:34)
>> summary(kiritimati_primary_roads)
>> td <- tempdir()
>> writeOGR(kiritimati_primary_roads, td, "kiri1", driver="ESRI Shapefile")
>> kiri1 <- readOGR(td, "kiri1")
>> summary(kiri1)
>> kiri1$dt <- as.POSIXct(as.character(kiri1$dt))
>> summary(kiri1)
>>
>> This failed in 0.7-1 because the POSIXct column is not recognized as also
>> POSIXt; this is corrected in 0.7-4. However, this was not the original
>> problem. Had you reported the output of:
>>
>> sapply(spdfUTM at data, class)
>> sapply(spdfUTM at data, storage.mode)
>>
>> we would have known how krillGMTtime was stored - it does not seem to
>> have
>> been detected as a column requiring conversion to character string, or as
>> a column that should trigger an error. In general, as Alex wrote, all
>> date
>> time columns should be converted to character and converted back on
>> reading back from the shapefile. OGR drivers treat field types
>> differently, so writeOGR has to take a conservative view of possibilities.
>
> Here's the output from the sapply:
>
>
>> sapply(spdfUTM at data, class)
> krillGMTtime          Lat          Lon
>    "factor"     "factor"     "factor"
>> sapply(spdfUTM at data, storage.mode)
> krillGMTtime          Lat          Lon
>   "integer"    "integer"    "integer"
>
>
> To answer Don's question, yes I have bad LAT/LON rows when I first read
> them in but they are all 999. I discard them before making my spatial
> points data frame, which does in fact use those two columns for the
> coordinates. I have another similar dataset where the date/time field
> storage.mode is reported as "character" and not "integer", and has the
> same issue.
>
>
> Thanks for the help.
>
> Corrie
>
>
>> Hope this clarifies,
>>
>> Roger
>>
>>> Thanks,
>>>
>>> Corrie
>>>
>>>
>>>
>>> On 11/11/11 5:31 PM, "Roger Bivand" <Roger.Bivand at nhh.no> wrote:
>>>
>>>> On Fri, 11 Nov 2011, Corrie Curtice wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Apologies if this has already been addressed somewhere, I did a brief
>>>>> search
>>>>> of archives but didn't find quite this issue.
>>>>
>>>> Which versions of what are you using? You should provide the output of
>>>> sessionInfo(), the messages given when rgdal loads, and details of how
>>>> you
>>>> installed rgdal. In addition, a use case would be helpful, but only
>>>> when
>>>> we know what version you are using. In current rgdal, data frame
>>>> columns
>>>> of class POSIXt are converted to character strings before writing with
>>>> writeOGR().
>>>>
>>>> Roger
>>>>
>>>>>
>>>>> I'm writing out an ESRI shapefile.  My spdf has a date/time field.
>>>>> Looks
>>>>> like this:
>>>>>
>>>>>> head(spdfUTM at data)
>>>>>           krillGMTtime          Lat          Lon
>>>>> 1   2010-05-12 12:34:21 -64.67655969 -62.15040195
>>>>> 21  2010-05-12 12:35:12  -64.6771229  -62.1519511
>>>>> 38  2010-05-12 12:36:02 -64.67775863 -62.15340614
>>>>> 57  2010-05-12 12:36:53 -64.67838269 -62.15494829
>>>>> 78  2010-05-12 12:37:43 -64.67901497 -62.15647203
>>>>> 100 2010-05-12 12:38:37 -64.67973667 -62.15773224
>>>>>
>>>>> All three fields are reported to be "factors" by str.  writeOGR
>>>>> appears
>>>>> happy:
>>>>>
>>>>>> writeOGR(spdfUTM,dd,layer="krillPoints-UTM",driver="ESRI
>>>>> Shapefile",verbose=TRUE,overwrite=TRUE)
>>>>> $object_type
>>>>> [1] "SpatialPointsDataFrame"
>>>>> $output_dsn
>>>>> [1] "/users/corriecurtice/documents/Data_2010/Shapefiles/"
>>>>> $output_layer
>>>>> [1] "krillPoints-UTM"
>>>>> $output_diver
>>>>> [1] "ESRI Shapefile"
>>>>> $output_n
>>>>> [1] 11179
>>>>> $output_nfields
>>>>> [1] 3
>>>>> $output_fields
>>>>> [1] "krillGMTtime" "Lat"          "Lon"
>>>>> $output_fclasses
>>>>> [1] 4 4 4
>>>>> $dataset_options
>>>>> NULL
>>>>> $layer_options
>>>>> NULL
>>>>> Warning message:
>>>>> In writeOGR(spdfUTM, dd, layer = "krillPoints-UTM", driver = "ESRI
>>>>> Shapefile",  :
>>>>>  existing layer removed
>>>>>
>>>>> When I read it right back in again, the date/time field is NAs.  This
>>>>> is
>>>>> also true if I load the shapefile into ArcMap.
>>>>>
>>>>>> foo <- readOGR(dd,layer="krillPoints-UTM",verbose=TRUE)
>>>>> OGR data source with driver: ESRI Shapefile
>>>>> Source: "/users/corriecurtice/documents/Data_2010/Shapefiles/", layer:
>>>>> "krillPoints-UTM"
>>>>> with 11179 features and 3 fields
>>>>> Feature type: wkbPoint with 2 dimensions
>>>>>
>>>>>> head(foo at data)
>>>>>  krillGMTti          Lat          Lon
>>>>> 1       <NA> -64.67655969 -62.15040195
>>>>> 2       <NA>  -64.6771229  -62.1519511
>>>>> 3       <NA> -64.67775863 -62.15340614
>>>>> 4       <NA> -64.67838269 -62.15494829
>>>>> 5       <NA> -64.67901497 -62.15647203
>>>>> 6       <NA> -64.67973667 -62.15773224
>>>>>
>>>>> Thoughts? Am I missing something obvious?
>>>>>
>>>>> I can break it up into separate date and time fields, but since the
>>>>> field is
>>>>> simply a factor anyways I'm not sure how that would help.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Corrie
>>>>>
>>>>> ---
>>>>> Corrie Curtice
>>>>> Research Analyst
>>>>> Marine Geospatial Ecology Lab
>>>>> Nicholas School of the Environment, Duke University
>>>>> http://mgel.env.duke.edu
>>>>> em: corrie.curtice at duke.edu
>>>>> ph: 252-504-7538
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>     [[alternative HTML version deleted]]
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-Geo mailing list
>>>>> R-sig-Geo at r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>
>>>>
>>>> --
>>>> Roger Bivand
>>>> Department of Economics, NHH Norwegian School of Economics,
>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>> voice: +47 55 95 93 55; fax +47 55 95 95 43
>>>> e-mail: Roger.Bivand at nhh.no
>>>>
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>> --
>> Roger Bivand
>> Department of Economics, NHH Norwegian School of Economics,
>> Helleveien 30, N-5045 Bergen, Norway.
>> voice: +47 55 95 93 55; fax +47 55 95 95 43
>> e-mail: Roger.Bivand at nhh.no
>
>
>

-- 
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no

More information about the R-sig-Geo mailing list