[R] using foreach (parallel processing)

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/02/2010 10:56 AM, Santosh Srinivas wrote:
> Hello group,

Hi

First of all: there is a special interest group for high performance
computing in R, which includes parallel computing, wghich would be more
appropriate to ask. But I will try anyway:

> 
> I am experimenting with parallel processing on my quad core Win 7 32
> bit machine. Using these packages for the first time.
> 
> I can see all my processor running at full performance when I use a
> smaller dataset
> 
> require(snow)
> require(doSNOW)
> require(foreach)
> #change the 8 to however many cores\phys processors you have on your machine
> cl.tmp = makeCluster(rep("localhost",4), type="SOCK")
> registerDoSNOW(cl.tmp)
> 
> 
> optData.df <- head(pristine,100000)
> 
> system.time(
> optData.df$impliedVol <-
> foreach(i=1:NROW(optData.df),.packages="RQuantLib") %dopar%
> with(optData.df[i,],
> 	tryCatch({EuropeanOptionImpliedVolatility(type,value,underlying,
> strike, dividendYield, riskFreeRate, maturity,
> volatility)$impliedVol},
> 		error = function (ex){0}))
> )
> 
> This works fine!
> 
> PROBLEM: However, when I do the same but with optData.df <- pristine
> ... which has about 3.8 million options data ... the cores do not seem
> to be fully utilized (they seem to run at 25%).
> 
> I noticed some slight delay before the processing starts running ...
> when I did with the 100k dataset ... do i need to wait longer for any
> allocations to be done?

Communication to setup the threads ould definitly take some time.
So why don't you try to increase from 100.000 to 1.000.000 and see how
long it takes to initialize the threads?

You are not mentioning how long you wait?

Cheers,

Rainer

> 
> Thank you.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Tel:        +33 - (0)9 53 10 27 44
Cell:       +27 - (0)8 39 47 90 42
Fax (SA):   +27 - (0)8 65 16 27 82
Fax (D) :   +49 - (0)3 21 21 25 22 44
Fax (FR):   +33 - (0)9 58 10 27 44
email:      Rainer at krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkz3b5YACgkQoYgNqgF2egrqtQCfQKXKZln5tz5eZDTYBmbFvruH
H/4AoIOXZJ7F3lKL6QLRkaQPHZbqBDQn
=odY/
-----END PGP SIGNATURE-----