[R-sig-hpc] Question

["Jens Oehlschlägel"]

Krystian,

You asked how to do

b=a[a>5]

with ff objects.


You can do that by storing the logical selection in a bit vector and then creating the target ff vector in the final size
library(ff)
n <- 4e7
s <- 1e3
a <- ff(1:n, caching="mmeachflush")
system.time({
  # creating the boolean vector
  f <- bit(n)
  for (i in chunk(a, by=s)) f[i] <- a[i]>5
  # now knowing the desires length create the target ff
  b <- clone(a, length=sum(f), update=FALSE)
  # fill it using the evaluated condition
  n1 <- 0L
  for (i in chunk(a, by=s)){
    h <- as.hi(f, range=i)
    n2 <- n1 + length(h)
    n1 <- n1 + 1L
    b[n1:n2] <- a[h]
    n1 <- n2
  }
})


However you can also save the RAM for the bit vector and - more important - a second pass over the input data by growing the target ff on the fly:

system.time({
  for (i in chunk(a, by=s))
  {
    x <- a[i]
    x <- x[x>5]
    if (i[[1]]==1L){
      b <- as.ff(x, caching="mmeachflush")
    }else{
      n1 <- length(b)
      n2 <- n1 + length(x)
      length(b) <- n2
      n1 <- n1 + 1L
      b[n1:n2] <- x
    }
  }
})


The latter for me performs better if the ff object are bigger than RAM, the former might be better if you want to combine several logical conditions.

Kind regards


Jens Oehlschlägel



-----Ursprüngliche Nachricht-----
Von: kradlak <krystian.radlak at comarch.pl>
Gesendet: 18.08.2011 11:24:21
An: Jens.Oehlschlaegel at truecluster.com
Betreff: Question

> Hi,
>I have a question connected to package ff.
>I want to do operation with ff vector from packages ff like simple
>operation in R like this.
>Example
>a=1:10
>b=a[a>5]
>
>How to do this operation with ff packages:
>Example :
>a=ff(1, length=15*10^7)
>
>I know that I could get logical vector like this:
>
>k=bit(15*10^7) # from bit packages
>a=ff(-1,15*10^7)
>system.time(
> for (i in chunk(a)){ k[i]<- a[i]<1}
>)
>
>Any suggestions?
>
>Best,
>Krystian