Dennis and all,


Thank you for the help as I try to get this method for importing and batch
processing files organized. I currently have this set-up to import data from
two files in my working directory. "Var1" specifies data from file 1 and
file 2.



filenames=list.files()

library(plyr)

import.list=adply(filenames, 1, read.csv)

import.list



   Var1 Time O2_conc Chla_conc

1     1    0   273.5       300

2     1   10   268.1       240

3     1   20   262.8       210

4     1   30   257.4       175

5     1   40   252.0       155

6     1   50   246.6       138

7     1   60   241.3       129

8     1   70   235.9       121

9     1   80   230.5       117

10    2    0   270.0       300

11    2   10   269.0       240

12    2   20   257.0       210

13    2   30   259.0       175

14    2   40   230.0       155

15    2   50   220.0       138

16    2   60   221.0       129

17    2   70   450.0       121

18    2   80   250.0       117



For my calculation I am only interested in column 3, so I used split() to
pull out this column based upon the value of Var1.





split(import.list[,3],import.list$Var1)

$`1`

[1] 273.5 268.1 262.8 257.4 252.0 246.6 241.3 235.9 230.5



$`2`

[1] 270 269 257 259 230 220 221 450 250



Finally I want to find the minimum value in column 3 (for each file) and the
three previous data points, and then calculate the mean of these four
values.



> a=which.min(data$"1")

> b=(which.min(data$"1")-3)

> c=c(a:b)

> d=mean(data$"1"[c])

> d

[1] 238.575



So far this works, but this last set of calculations (the min, mean) are not
very automated and are probably a bit cumbersome.

Rather than having to replace a=which.min(data$”1”) with  a=which.min(data$”2”)
to do this calculation on the second file I would like to write a loop or
use some R package with looping abilities to cycle through this (I have more
than 2 files, this is just a test set).



I’m imagining a psuedocode something like this



For (i in 1:20) {

a[i]=which.min(data$”[i]”)

b[i]=which.min(data$”[i]”)-3

c[i]=c(a[i]:b[i])

etc…

 ...though this particular code is most certainly too cumbersome.



Any thoughts on how to make a loop so that I can cycle through the files and
run these calculations?



Thank you,

Nate


On Mon, Nov 15, 2010 at 9:12 PM, Dennis Murphy <djmuser@gmail.com> wrote:

> Hi:
>
> See inline.
>
> On Mon, Nov 15, 2010 at 4:26 PM, Nate Miller <natemiller77@gmail.com>wrote:
>
>> Hi All!
>>
>> I have some experience with R, but less experience writing scripts using
>> R and have run into a challenge that I hope someone can help me with.
>>
>>  I have multiple .csv files of data each with the same 3 columns of
>> data, but potentially of varying lengths (some data files are from short
>> measurements, others from longer ones). One file for example might look
>> like this...
>>
>> Time, O2_conc, Chla_conc
>>
>> 0,270,300
>>
>> 10, 260, 280
>>
>> 20, 245, 268
>>
>> 30, 233, 238
>>
>> 40, 222, 212
>>
>> 50, 215, 201
>>
>> 60, 208, 193
>>
>> 70, 206, 191
>>
>> 80, 207,189
>>
>> 90, 206, 186
>>
>> 100, 206, 183
>>
>> 110, 207, 178
>>
>> 120, 205, 174
>>
>> 130, 240, 171
>>
>> 140, 270, 155
>>
>> I am looking for an efficient means of batch (or sequentially)
>> processing these files so that I can
>> 1. import each data file
>>
>> 2. find the minimum value recorded in column 2 and the previous 5 data
>> points
>>
>
> Don't know what you mean by the 'previous 5 data points' ...are you
> referring to a rolling minimum?
>
>>
>> 3. and average these 10 values to get a mean, minimum value.
>>
>
> If the surmise above is correct, you should get 11 rolling mins for a
> vector of length 15. Here's an example using the rollapply() function from
> the zoo package:
>
> library(zoo)
> > x <- rpois(15, 10)
> > x
>  [1] 17 12 17  9  8 10  7 11 15 11 11 15  5  9 12
> > rollapply(zoo(x), 5, FUN = min)
>  3  4  5  6  7  8  9 10 11 12 13
>  8  8  7  7  7  7  7 11  5  5  5
> > mean(rollapply(zoo(x), 5, FUN = min))
> [1] 7
>
>
> Currently I have imported the data files using the following
>>
>> filenames=list.files()
>>
>> library(plyr)
>>
>> import.list=adply(filenames, 1, read.csv)
>>
>
> This seems to be a reasonable approach. Does the result keep a column for
> the file names?
>
>>
>> and I know how to write a code to calculate the minimum value and the 5
>> preceding values in a single column, in a single file. I think the
>> problem I am running into is scaling this code up so that I can import
>> multiple files and calculating mean, minimum value for the 2^nd column
>> in each of them.
>>
>
> As long as you have an indicator for each file, this should be pretty
> straightforward. Write a function that produces the summaries for one data
> frame and then do something like
>
> ddply(slurpedFiles, .(dsIndicator), myfunction)
>
> to map the function to all of them.
>
> The data.table package is an alternative, where you should be able to do
> similar things using the data set names as a key. data.table 'thinks' more
> like an SQL, but it can be very efficient.
>
> You should be able to do what you're asking for with either package.
>
> HTH,
> Dennis
>
>>
>> Can anyone offer some advice on how to batch processes a whole bunch of
>> files? I need to load them in, but then analyze them too.
>>
>> Thank you so much,
>>
>> Nate
>>
>>
>>        [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>

	[[alternative HTML version deleted]]

