This may or may not help, but for truly independent calculations (e.g.
reading and normalizing a pile of arrays) I find that writing a function to
do the task end-to-end and then handing it off to mclapply(list.of.keys,
function) typically results in a near-linear speedup.  However, multicore is
really not the most elegant way to do that sort of thing.  If you look at
what Benilton Carvalho has done in the 'oligo' package, you will see a far
more memory-efficient approach using the 'ff' and 'bit' packages to share a
(supercached) flat-file image to successively stride through the chunks of
data.

Anyways, I'm dumb so I just use mclapply() and keep my memory image small,
run gc() a lot, and mull over using 'oligo'.



On Mon, Oct 18, 2010 at 9:05 AM, Edwin Groot <
edwin.groot@biologie.uni-freiburg.de> wrote:

> Hello all,
> I have difficulty getting the multicore package doing what it promises.
> Does anybody have a benchmark that demonstrates something intensive
> with and without multicore assistance?
> I have a dual dual-core Xeon, and $ top tells me all R can squeeze from
> my Linux system is 25% us. Here is my example:
>
> > library(Starr)
> #Read in a set of ChIP-chip arrays
> > read("array.rda")
> # $ top reports 25% us for the following:
> > array_norm <- normalize.Probes(array, method = "loess")
> #Try the same with multicore
> > library(multicore)
> > multicore:::detectCores()
> [1] 4
> #No benefit from multicore. $ top reports 25% us for the following:
> > array_norm <- normalize.Probes(array, method = "loess")
> #lattice masks out parallel from multicore. Use mcparallel instead.
> > pnorm <- mcparallel(normalize.Probes(array, method = "loess"))
> > Normalizing probes with method: loess
> Done with 1 vs 2 in iteration 1
> #Function continues for some time and displays more messages. No
> benefit from multicore. $ top reports 25% us during the run...
> > array_norm <- collect(pnorm)
> #Oh dear, where did my normalized data go?
> > array_norm
> $`4037`
> NULL
> > sessionInfo()
> R version 2.11.1 (2010-05-31)
> x86_64-pc-linux-gnu
>
> locale:
>  [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C
>  [3] LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8
>  [5] LC_MONETARY=C              LC_MESSAGES=en_GB.UTF-8
>  [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C
>  [9] LC_ADDRESS=C               LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] tools     grid      stats     graphics  grDevices utils
>    datasets
> [8] methods   base
>
> other attached packages:
>  [1] geneplotter_1.26.0   annotate_1.26.1      AnnotationDbi_1.10.2
>  [4] Starr_1.4.4          affxparser_1.20.0    affy_1.26.1
>  [7] Ringo_1.12.0         Matrix_0.999375-39   lattice_0.18-8
> [10] limma_3.4.4          RColorBrewer_1.0-2   Biobase_2.8.0
> [13] multicore_0.1-3
>
> loaded via a namespace (and not attached):
>  [1] affyio_1.16.0         DBI_0.2-5             genefilter_1.30.0
>  [4] MASS_7.3-6            preprocessCore_1.10.0 pspline_1.0-14
>  [7] RSQLite_0.9-2         splines_2.11.1        survival_2.35-8
> [10] tcltk_2.11.1          xtable_1.5-6
>
> RTFMing only gives me the syntax of some functions in the multicore
> package. How do I apply successfully this thing to my code?
>
> Regards,
> Edwin
> --
> Dr. Edwin Groot, postdoctoral associate
> AG Laux
> Institut fuer Biologie III
> Schaenzlestr. 1
> 79104 Freiburg, Deutschland
> +49 761-2032945
>
> _______________________________________________
> Bioconductor mailing list
> Bioconductor@stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/bioconductor
> Search the archives:
> http://news.gmane.org/gmane.science.biology.informatics.conductor
>



-- 
With four parameters I can fit an elephant, and with five I can make him
wiggle his trunk.

John von Neumann

	[[alternative HTML version deleted]]

