# [R] How to speed up interpolation

jim holtman jholtman at gmail.com
Mon Jul 18 04:11:27 CEST 2011

```Here is what I did; convert the data to a numeric matrix for faster
processing.  You can convert back to a dataframe since you have the
indices into the levels for the flights and runways.

> source('/temp/df/df')
> # convert to matrix
> df.mat <- cbind(pt = as.numeric(df\$PredTime)
+               , dt = as.numeric(df\$dt)
+               , rw = as.numeric(df\$lrw)  # index into 'levels'
+               , flight = as.numeric(df\$flightfact)
+               )
> # create a list of row numbers for each flight for processing
> flgt.list <- split(seq(nrow(df.mat)), df.mat[, 'flight'])
> # remove lists with only 1 entry
> flgt.list <- flgt.list[sapply(flgt.list, length) > 1]
>
> # create the interval we want data for
> interval <- as.numeric(0:60)
>
> # now process the flights
> times <- lapply(flgt.list, function(.flt){
+     interp <- approx(df.mat[.flt, 'pt']
+                     , df.mat[.flt, 'dt']
+                     , xout = interval
+                     , rule = 1
+                     )
+     # return vector
+     cbind(time = interp\$x
+       , error = interp\$y
+       , runway = df.mat[.flt[1L], 'rw']
+       , flight = df.mat[.flt[1L], 'flight']
+       )
+ })
> # sample output -- is this correct?
> times[[1]]
time       error runway flight
[1,]    0          NA      2      1
[2,]    1          NA      2      1
[3,]    2 -0.13795380      2      1
[4,]    3 -0.20726073      2      1
[5,]    4 -0.27309237      2      1
[6,]    5 -0.33333333      2      1
[7,]    6 -0.09322419      2      1
[8,]    7  0.14688495      2      1
[9,]    8  0.38699409      2      1
[10,]    9  0.62710323      2      1
[11,]   10  0.86721237      2      1
[12,]   11  1.10732151      2      1
[13,]   12  1.34743065      2      1
[14,]   13  1.58753979      2      1
[15,]   14  1.82764893      2      1
[16,]   15  2.06775807      2      1
[17,]   16  2.30786721      2      1
[18,]   17  2.54797635      2      1
[19,]   18  6.66600000      2      1
[20,]   19  4.82600000      2      1
[21,]   20  3.00436508      2      1
[22,]   21  2.22316562      2      1
[23,]   22  1.34895178      2      1
[24,]   23  0.47473795      2      1
[25,]   24 -0.39947589      2      1
[26,]   25 -1.27368973      2      1
[27,]   26 -2.12478632      2      1
[28,]   27 -1.61196581      2      1
[29,]   28 -1.09914530      2      1
[30,]   29 -0.58632479      2      1
[31,]   30 -0.07350427      2      1
[32,]   31  0.43931624      2      1
[33,]   32  0.95213675      2      1
[34,]   33  1.46495726      2      1
[35,]   34  1.97777778      2      1
[36,]   35  2.49059829      2      1
[37,]   36  3.00341880      2      1
[38,]   37  3.51623932      2      1
[39,]   38  4.02905983      2      1
[40,]   39  4.54188034      2      1
[41,]   40  5.05470085      2      1
[42,]   41  5.53360434      2      1
[43,]   42  5.53766938      2      1
[44,]   43  5.54173442      2      1
[45,]   44  5.54579946      2      1
[46,]   45  5.54986450      2      1
[47,]   46  5.55392954      2      1
[48,]   47  5.55799458      2      1
[49,]   48  5.56205962      2      1
[50,]   49  5.56612466      2      1
[51,]   50  5.57018970      2      1
[52,]   51  5.57425474      2      1
[53,]   52  5.57831978      2      1
[54,]   53  5.58238482      2      1
[55,]   54  5.58644986      2      1
[56,]   55  5.59051491      2      1
[57,]   56  5.59457995      2      1
[58,]   57  5.59864499      2      1
[59,]   58  5.60271003      2      1
[60,]   59  5.60677507      2      1
[61,]   60  5.61084011      2      1

On Sun, Jul 17, 2011 at 6:58 PM, James Rome <jamesrome at gmail.com> wrote:
> I thought I had included the data... Here it is again.
>
> What I want to do is to make box and whisker plots with each flight
> counted the same number of times in each time bin. Hence the
> interpolation to minute time hacks.
>
>
> On 7/17/2011 4:16 PM, jim holtman wrote:
>> It would be nice if you had some sample data included so that we could
>> see how the code worked.  Have you use Rprof on the code to see where
>> you are spending your time?  You might want to use 'matrix' instead of
>> 'data.frames' since there is a big performance impact with dataframes
>> when indexing.  A little more description of the problem you are
>> trying to solve would also be useful.  I tend to ask people "tell me
>> what you want to do, not how you want to do it".
>>
>> On Sun, Jul 17, 2011 at 1:30 PM, James Rome <jamesrome at gmail.com> wrote:
>>> df is a very large data frame with arrival estimates for many flights
>>> (DF\$flightfact) at random times (df\$PredTime). The error of the estimate
>>> is df\$dt.
>>> My problem is that I want to know the prediction error at each minute
>>> before landing. This code works, but is very slow, and dominates
>>> everything. I tried using split(), but that rapidly ate up my 12 GB of
>>> memory. So, is there a better R way of doing this?
>>>
>>> Thanks,
>>> Jim Rome
>>>
>>>    flights = table(df\$flightfact[1:dim(df)[1], drop=TRUE])
>>>    nflights = length(flights)
>>>    flights = as.data.frame(flights)
>>>    times = data.frame()
>>>    # Split by flight
>>>    for(i in 1:nflights) {
>>>        tf = df[as.numeric(df\$flightfact)==flights[i,1],]    # This flight
>>>        #check for at least 2 entries
>>>        if(dim(tf)[1] < 2) {
>>>            next
>>>        }
>>>        idf = interpolateTimes(tf)
>>>        times = rbind(times, idf)
>>>    }
>>>
>>> # Interpolate the times to every minute for 60 minutes
>>> # Return a new data frame
>>> interpolateTimes = function(df) {
>>>   x = as.numeric(seq(from=0,to=60)) # The times to interpolate to
>>>   dti = approx(as.numeric(df\$PredTime), as.numeric(df\$dt), x,
>>> method="linear",rule=1:1)
>>>   # Make a new data frame of interpolated values
>>>   idf = data.frame(time=dti\$x, error=dti\$y,
>>>       runway=rep(df\$lrw[1],length(dti\$x)),
>>> flight=rep(df\$flightfact[1], length(dti\$x)))
>>>   return(idf)
>>> }
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>>
>>
>>
>
>

--
Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?

```