[R] using foreach (parallel processing)

[Santosh Srinivas]

Thanks. I guess I had to wait long enough to get the threads running
and now they seem be on top gear.
Hope this thing doesn't crash now!


On Thu, Dec 2, 2010 at 3:36 PM, Rainer M Krug <r.m.krug at gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 12/02/2010 10:56 AM, Santosh Srinivas wrote:
>> Hello group,
>
> Hi
>
> First of all: there is a special interest group for high performance
> computing in R, which includes parallel computing, wghich would be more
> appropriate to ask. But I will try anyway:
>
>>
>> I am experimenting with parallel processing on my quad core Win 7 32
>> bit machine. Using these packages for the first time.
>>
>> I can see all my processor running at full performance when I use a
>> smaller dataset
>>
>> require(snow)
>> require(doSNOW)
>> require(foreach)
>> #change the 8 to however many cores\phys processors you have on your machine
>> cl.tmp = makeCluster(rep("localhost",4), type="SOCK")
>> registerDoSNOW(cl.tmp)
>>
>>
>> optData.df <- head(pristine,100000)
>>
>> system.time(
>> optData.df$impliedVol <-
>> foreach(i=1:NROW(optData.df),.packages="RQuantLib") %dopar%
>> with(optData.df[i,],
>>       tryCatch({EuropeanOptionImpliedVolatility(type,value,underlying,
>> strike, dividendYield, riskFreeRate, maturity,
>> volatility)$impliedVol},
>>               error = function (ex){0}))
>> )
>>
>> This works fine!
>>
>> PROBLEM: However, when I do the same but with optData.df <- pristine
>> ... which has about 3.8 million options data ... the cores do not seem
>> to be fully utilized (they seem to run at 25%).
>>
>> I noticed some slight delay before the processing starts running ...
>> when I did with the 100k dataset ... do i need to wait longer for any
>> allocations to be done?
>
> Communication to setup the threads ould definitly take some time.
> So why don't you try to increase from 100.000 to 1.000.000 and see how
> long it takes to initialize the threads?
>
> You are not mentioning how long you wait?
>
> Cheers,
>
> Rainer
>
>>
>> Thank you.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>
> - --
> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
> Biology, UCT), Dipl. Phys. (Germany)
>
> Centre of Excellence for Invasion Biology
> Natural Sciences Building
> Office Suite 2039
> Stellenbosch University
> Main Campus, Merriman Avenue
> Stellenbosch
> South Africa
>
> Tel:        +33 - (0)9 53 10 27 44
> Cell:       +27 - (0)8 39 47 90 42
> Fax (SA):   +27 - (0)8 65 16 27 82
> Fax (D) :   +49 - (0)3 21 21 25 22 44
> Fax (FR):   +33 - (0)9 58 10 27 44
> email:      Rainer at krugs.de
>
> Skype:      RMkrug
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkz3b5YACgkQoYgNqgF2egrqtQCfQKXKZln5tz5eZDTYBmbFvruH
> H/4AoIOXZJ7F3lKL6QLRkaQPHZbqBDQn
> =odY/
> -----END PGP SIGNATURE-----
>