[R] reading data from web data sources
tim+r-project.org at coote.org
Sat Feb 27 21:28:36 CET 2010
Thanks, Gabor. My take away from this and Phil's post is that I'm
going to have to construct some code to do the parsing, rather than
use a standard function. I'm afraid that neither approach works, yet:
Gabor's gets has an off-by-one error (days start on the 2nd, not the
first), and the years get messed up around the 29th day. I think that
na.omit (DF) line is throwing out the baby with the bathwater. It's
interesting that this approach is based on read.table, I'd assumed
that I'd need read.ftable, which I couldn't understand the
documentation for. What is it that's removing the -999 and -888
values in this code -they seem to be gone, but I cannot see why.
Phil's reads in the data, but interleaves rows with just a year and
all other values as NA.
On 27 Feb 2010, at 17:33, Gabor Grothendieck wrote:
> Mark Leeds pointed out to me that the code wrapped around in the post
> so it may not be obvious that the regular expression in the grep is
> (i.e. it contains a space):
> "[^ 0-9.]"
> On Sat, Feb 27, 2010 at 7:15 AM, Gabor Grothendieck
> <ggrothendieck at gmail.com> wrote:
>> Try this. First we read the raw lines into R using grep to remove
>> lines containing a character that is not a number or space. Then we
>> look for the year lines and repeat them down V1 using cumsum.
>> we omit the year lines.
>> myURL <- "http://climate.arm.ac.uk/calibrated/soil/dsoil100_cal_1910-1919.dat
>> raw.lines <- readLines(myURL)
>> DF <- read.table(textConnection(raw.lines[!grepl("[^
>> 0-9.]",raw.lines)]), fill = TRUE)
>> DF$V1 <- DF[cumsum(is.na(DF[])), 1]
>> DF <- na.omit(DF)
>> On Sat, Feb 27, 2010 at 6:32 AM, Tim Coote <tim+r-project.org at coote.org
>> > wrote:
>>> I'm trying to read some time series data of meteorological records
>>> that are
>>> available on the web (eg
>>> dsoil100_cal_1910-1919.dat). I'd
>>> like to be able to read in the digital data directly into R.
>>> However, I
>>> cannot work out the right function and set of parameters to use.
>>> It could
>>> be that the only practical route is to write a parser, possibly in
>>> other language, reformat the files and then read these into R. As
>>> far as I
>>> can tell, the informal grammar of the file is:
>>> <comments terminated by a blank line>
>>> [<year number on a line on its own>
>>> <daily readings lines> ]+
>>> and the <daily readings> are of the form:
>>> <whitespace> <day number> [<whitespace> <reading on day of month>]
>>> Readings for days in months where a day does not exist have
>>> special values.
>>> Missing values have a different special value.
>>> And then I've got the problem of iterating over all relevant files
>>> to get a
>>> whole timeseries.
>>> Is there a way to read in this type of file into R? I've read all
>>> of the
>>> examples that I can find, but cannot work out how to do it. I
>>> don't think
>>> that read.table can handle the separate sections of data
>>> representing each
>>> year. read.ftable maybe can be coerced to parse the data, but I
>>> cannot see
>>> how after reading the documentation and experimenting with the
>>> I'm using R 2.10.1 on osx 10.5.8 and 2.10.0 on Fedora 10.
>>> Any help/suggestions would be greatly appreciated. I can see that
>>> this type
>>> of issue is likely to grow in importance, and I'd also like to
>>> give the data
>>> owners suggestions on how to reformat their data so that it is
>>> easier to
>>> consume by machines, while being easy to read for humans.
>>> The early records are a serious machine parsing challenge as they
>>> are tiff
>>> images of old notebooks ;-)
>>> Tim Coote
>>> tim at coote.org
>>> vincit veritas
>>> R-help at r-project.org mailing list
>>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
tim at coote.org
More information about the R-help