[R-sig-hpc] bugs or strange interactions between multicore and ff

[Ramon Diaz-Uriarte]

Dear All,

I am observing strange interactions between the parallel function
(library multicore) and library ff: the objects written by ff are
wrong.

- The problem shows up when the function passed to multicore produces
printed output (e.g., "print" or "cat") AND multicore is called with
argument "silent = TRUE".

- The problem can be avoided if ff is forced to release the memory
resources (using "close(ffobject)") BEFORE the print statement. See
the difference between "my.f3" and "my.f4" below.

So it seems that standard output is getting mixed with the memory
mapped by ff before the cache is flushed.


I am not sure if the bug is in "ff", or in "multicore", or in my usage
of the combination.


Code with reproducible examples follows. I see the problems in at
least three different systems (all GNU/Linux, both i686 and x86-64; R
2.10 patched --different versions--; latest versions of multicore and
ff). Version information at end.

Best,

R.



##### Examples

## Beware: running the examples will leave a bunch of files named
bugexample*.ff
## under getwd()
## vmode used does not matter; problems are seen with any vmode
--though output differs

library(ff)

my.f1 <- function() { ## problems
  vv <- ff(1:5, vmode = "integer", pattern = paste(getwd(),
"/bugexample", sep = ""))
  save(file = "vv.RData", vv)
  print(2 * 2)
}

my.f2 <- function() { ## OK
  vv <- ff(1:5, vmode = "integer", pattern = paste(getwd(),
"/bugexample", sep = ""))
  close(vv)
  save(file = "vv.RData", vv)
  print(2 * 2)
}

my.f3 <- function() { ## OK
  vv <- ff(1:5, vmode = "integer", pattern = paste(getwd(),
"/bugexample", sep = ""))
  save(file = "vv.RData", vv)
  close(vv)
  print(2 * 2)
}

my.f4 <- function() { ## problems
  vv <- ff(1:5, vmode = "integer", pattern = paste(getwd(),
"/bugexample", sep = ""))
  save(file = "vv.RData", vv)
  print(2 * 2)
  close(vv)
}


my.f5 <- function() { ## OK
  vv <- ff(1:5, vmode = "integer", pattern = paste(getwd(),
"/bugexample", sep = ""))
  save(file = "vv.RData", vv)
}


library(multicore)

## Problems
parallel(my.f1(), silent = TRUE)
collect()
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)

## OK
parallel(my.f2(), silent = TRUE)
collect()
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)

## OK
parallel(my.f3(), silent = TRUE)
collect()
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)


## Problems
parallel(my.f4(), silent = TRUE)
collect()
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)

## OK
parallel(my.f5(), silent = TRUE)
collect()
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)


# OK
parallel(my.f1(), silent = FALSE)
collect()
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)




### Library fork works OK

library(fork)
## OK
pid <- fork(my.f1); wait(pid)
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)

## OK
pid <- fork(my.f2); wait(pid)
load("vv.RData"); open(vv); filename(vv); vv; rm(vv)



#################



Version information: system 1:

> version
               _
platform       i686-pc-linux-gnu
arch           i686
os             linux-gnu
system         i686, linux-gnu
status         Patched
major          2
minor          10.0
year           2009
month          11
day            04
svn rev        50311
language       R
version.string R version 2.10.0 Patched (2009-11-04 r50311)
> packageDescription("multicore")$Version
[1] "0.1-3"
> packageDescription("ff")$Version
[1] "2.1-1"


Version information: system 2:
> version
               _
platform       x86_64-unknown-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status         Patched
major          2
minor          10.0
year           2009
month          11
day            05
svn rev        50317
language       R
version.string R version 2.10.0 Patched (2009-11-05 r50317)

> packageDescription("multicore")$Version
[1] "0.1-3"

> packageDescription("ff")$Version
[1] "2.1-1"



Version information: system 3:
> version
               _
platform       x86_64-unknown-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status         Patched
major          2
minor          10.0
year           2009
month          11
day            09
svn rev        50375
language       R
version.string R version 2.10.0 Patched (2009-11-09 r50375)
> packageDescription("multicore")$Version
[1] "0.1-3"
> packageDescription("ff")$Version
[1] "2.1-1"





-- 
Ramon Diaz-Uriarte
Structural Biology and Biocomputing Programme
Spanish National Cancer Centre (CNIO)
http://ligarto.org/rdiaz
Phone: +34-91-732-8000 ext. 3019