

Hello,


i'm pretty confused. I want to speed up my algorithm by using mclapply:
parallel, but when I compare time efficiency, apply still wins.

I'm smoothing log2ratio data by rq.fit.fnb:quantreg which is called by my 
function quantsm and I'm wrapping my data into matrix/list for apply/lapply
(mclapply) usage.




I adjust my data like this:

<code><span class='pln'>q </span><span class='pun'>=</span><span class='pln'> 
matrix</span><span class='pun'>(</span><span class='pln'>data</span><span 
class='pun'>,</span><span class='pln'> ncol</span><span 
class='pun'>=</span><span class='pln'>N</span><span class='pun'>)</span><span 
class='pln'>        </span><span class='com'># wrapping into matrix (using N =
 2, 4, 6 or 8)</span><span class='pln'>
ql </span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='pln'>list</span><span class='pun'>(</span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='pln'>data</span><span class='pun'>.</span><span 
class='pln'>frame</span><span class='pun'>(</span><span 
class='pln'>q</span><span class='pun'>))</span><span class='pln'>  
</span><span class='com'># making list</span></code>

And time comparing:

<code><span class='pln'>apply</span><span class='pun'>=</span><span 
class='pln'>system</span><span class='pun'>.</span><span 
class='pln'>time</span><span class='pun'>(</span><span 
class='pln'>apply</span><span class='pun'>(</span><span 
class='pln'>q</span><span class='pun'>,</span><span class='pln'> </span><span 
class='lit'>1</span><span class='pun'>,</span><span class='pln'> 
FUN</span><span class='pun'>=</span><span class='pln'>quantsm</span><span 
class='pun'>,</span><span class='pln'> </span><span 
class='lit'>0.50</span><span class='pun'>,</span><span class='pln'> 
</span><span class='lit'>2</span><span class='pun'>))</span><span class='pln'>
lapply</span><span class='pun'>=</span><span class='pln'>system</span><span 
class='pun'>.</span><span class='pln'>time</span><span 
class='pun'>(</span><span class='pln'>lapply</span><span 
class='pun'>(</span><span class='pln'>ql</span><span class='pun'>,</span><span
 class='pln'> FUN</span><span class='pun'>=</span><span 
class='pln'>quantsm</span><span class='pun'>,</span><span class='pln'> 
</span><span class='lit'>0.50</span><span class='pun'>,</span><span 
class='pln'> </span><span class='lit'>2</span><span class='pun'>))</span><span
 class='pln'>
mc2lapply</span><span class='pun'>=</span><span class='pln'>system</span><span
 class='pun'>.</span><span class='pln'>time</span><span 
class='pun'>(</span><span class='pln'>mclapply</span><span 
class='pun'>(</span><span class='pln'>ql</span><span class='pun'>,</span><span
 class='pln'> FUN</span><span class='pun'>=</span><span 
class='pln'>quantsm</span><span class='pun'>,</span><span class='pln'> 
</span><span class='lit'>0.50</span><span class='pun'>,</span><span 
class='pln'> </span><span class='lit'>2</span><span class='pun'>,</span><span 
class='pln'> mc</span><span class='pun'>.</span><span 
class='pln'>cores</span><span class='pun'>=</span><span 
class='lit'>2</span><span class='pun'>))</span><span class='pln'>
mc4lapply</span><span class='pun'>=</span><span class='pln'>system</span><span
 class='pun'>.</span><span class='pln'>time</span><span 
class='pun'>(</span><span class='pln'>mclapply</span><span 
class='pun'>(</span><span class='pln'>ql</span><span class='pun'>,</span><span
 class='pln'> FUN</span><span class='pun'>=</span><span 
class='pln'>quantsm</span><span class='pun'>,</span><span class='pln'> 
</span><span class='lit'>0.50</span><span class='pun'>,</span><span 
class='pln'> </span><span class='lit'>2</span><span class='pun'>,</span><span 
class='pln'> mc</span><span class='pun'>.</span><span 
class='pln'>cores</span><span class='pun'>=</span><span 
class='lit'>4</span><span class='pun'>))</span><span class='pln'>
mc6lapply</span><span class='pun'>=</span><span class='pln'>system</span><span
 class='pun'>.</span><span class='pln'>time</span><span 
class='pun'>(</span><span class='pln'>mclapply</span><span 
class='pun'>(</span><span class='pln'>ql</span><span class='pun'>,</span><span
 class='pln'> FUN</span><span class='pun'>=</span><span 
class='pln'>quantsm</span><span class='pun'>,</span><span class='pln'> 
</span><span class='lit'>0.50</span><span class='pun'>,</span><span 
class='pln'> </span><span class='lit'>2</span><span class='pun'>,</span><span 
class='pln'> mc</span><span class='pun'>.</span><span 
class='pln'>cores</span><span class='pun'>=</span><span 
class='lit'>6</span><span class='pun'>))</span><span class='pln'>
mc8lapply</span><span class='pun'>=</span><span class='pln'>system</span><span
 class='pun'>.</span><span class='pln'>time</span><span 
class='pun'>(</span><span class='pln'>mclapply</span><span 
class='pun'>(</span><span class='pln'>ql</span><span class='pun'>,</span><span
 class='pln'> FUN</span><span class='pun'>=</span><span 
class='pln'>quantsm</span><span class='pun'>,</span><span class='pln'> 
</span><span class='lit'>0.50</span><span class='pun'>,</span><span 
class='pln'> </span><span class='lit'>2</span><span class='pun'>,</span><span 
class='pln'> mc</span><span class='pun'>.</span><span 
class='pln'>cores</span><span class='pun'>=</span><span 
class='lit'>8</span><span class='pun'>))</span><span class='pln'>
timing</span><span class='pun'>=</span><span class='pln'>rbind</span><span 
class='pun'>(</span><span class='pln'>apply</span><span 
class='pun'>,</span><span class='pln'>lapply</span><span 
class='pun'>,</span><span class='pln'>mc2lapply</span><span 
class='pun'>,</span><span class='pln'>mc4lapply</span><span 
class='pun'>,</span><span class='pln'>mc6lapply</span><span 
class='pun'>,</span><span class='pln'>mc8lapply</span><span 
class='pun'>)</span></code>

Function quantsm:

<code><span class='pln'>quantsm </span><span class='pun'><-</span><span 
class='pln'> </span><span class='kwd'>function</span><span class='pln'> 
</span><span class='pun'>(</span><span class='pln'>y</span><span 
class='pun'>,</span><span class='pln'> p </span><span 
class='pun'>=</span><span class='pln'> </span><span 
class='lit'>0.5</span><span class='pun'>,</span><span class='pln'> 
</span><span class='kwd'>lambda</span><span class='pun'>)</span><span 
class='pln'> </span><span class='pun'>{</span><span class='pln'>
   </span><span class='com'># Quantile smoothing</span><span class='pln'>
   </span><span class='com'># Input: response y, quantile level p (0<p<1), 
smoothing parmeter lambda</span><span class='pln'>
   </span><span class='com'># Result: quantile curve</span><span class='pln'>

   </span><span class='com'># Augment the data for the difference 
penalty</span><span class='pln'>
   m </span><span class='pun'><-</span><span class='pln'> length</span><span 
class='pun'>(</span><span class='pln'>y</span><span class='pun'>)</span><span 
class='pln'>
   E </span><span class='pun'><-</span><span class='pln'> diag</span><span 
class='pun'>(</span><span class='pln'>m</span><span class='pun'>);</span><span
 class='pln'>
   </span><span class='typ'>Dmat</span><span class='pln'> </span><span 
class='pun'><-</span><span class='pln'> diff</span><span 
class='pun'>(</span><span class='pln'>E</span><span class='pun'>);</span><span
 class='pln'>
   X </span><span class='pun'><-</span><span class='pln'> rbind</span><span 
class='pun'>(</span><span class='pln'>E</span><span class='pun'>,</span><span 
class='pln'> </span><span class='kwd'>lambda</span><span class='pln'> 
</span><span class='pun'>*</span><span class='pln'> </span><span 
class='typ'>Dmat</span><span class='pun'>)</span><span class='pln'>
   u </span><span class='pun'><-</span><span class='pln'> c</span><span 
class='pun'>(</span><span class='pln'>y</span><span class='pun'>,</span><span 
class='pln'> rep</span><span class='pun'>(</span><span 
class='lit'>0</span><span class='pun'>,</span><span class='pln'> m 
</span><span class='pun'>-</span><span class='pln'> </span><span 
class='lit'>1</span><span class='pun'>))</span><span class='pln'>

   </span><span class='com'># Call quantile regression</span><span 
class='pln'>
   q </span><span class='pun'><-</span><span class='pln'> rq</span><span 
class='pun'>.</span><span class='pln'>fit</span><span 
class='pun'>.</span><span class='pln'>fnb</span><span 
class='pun'>(</span><span class='pln'>X</span><span class='pun'>,</span><span 
class='pln'> u</span><span class='pun'>,</span><span class='pln'> tau 
</span><span class='pun'>=</span><span class='pln'> p</span><span 
class='pun'>)</span><span class='pln'>
   q
</span><span class='pun'>}</span></code>

Function rq.fit.fnb (quantreg library):

<code><span class='pln'>rq</span><span class='pun'>.</span><span 
class='pln'>fit</span><span class='pun'>.</span><span class='pln'>fnb 
</span><span class='pun'><-</span><span class='pln'> </span><span 
class='kwd'>function</span><span class='pln'> </span><span 
class='pun'>(</span><span class='pln'>x</span><span class='pun'>,</span><span 
class='pln'> y</span><span class='pun'>,</span><span class='pln'> tau 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='lit'>0.5</span><span class='pun'>,</span><span class='pln'> beta 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='lit'>0.99995</span><span class='pun'>,</span><span class='pln'> eps 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='lit'>1e-06</span><span class='pun'>)</span><span class='pln'> 
</span><span class='pun'>{</span><span class='pln'>
    n </span><span class='pun'><-</span><span class='pln'> length</span><span 
class='pun'>(</span><span class='pln'>y</span><span class='pun'>)</span><span 
class='pln'>
    p </span><span class='pun'><-</span><span class='pln'> ncol</span><span 
class='pun'>(</span><span class='pln'>x</span><span class='pun'>)</span><span 
class='pln'>
    </span><span class='kwd'>if</span><span class='pln'> </span><span 
class='pun'>(</span><span class='pln'>n </span><span 
class='pun'>!=</span><span class='pln'> nrow</span><span 
class='pun'>(</span><span class='pln'>x</span><span class='pun'>))</span><span
 class='pln'> 
        stop</span><span class='pun'>(</span><span class='str'>"x and y don't 
match n"</span><span class='pun'>)</span><span class='pln'>
    </span><span class='kwd'>if</span><span class='pln'> </span><span 
class='pun'>(</span><span class='pln'>tau </span><span 
class='pun'><</span><span class='pln'> eps </span><span 
class='pun'>||</span><span class='pln'> tau </span><span 
class='pun'>></span><span class='pln'> </span><span class='lit'>1</span><span 
class='pln'> </span><span class='pun'>-</span><span class='pln'> 
eps</span><span class='pun'>)</span><span class='pln'> 
        stop</span><span class='pun'>(</span><span class='str'>"No parametric 
Frisch-Newton method.  Set tau in (0,1)"</span><span class='pun'>)</span><span
 class='pln'>
    rhs </span><span class='pun'><-</span><span class='pln'> </span><span 
class='pun'>(</span><span class='lit'>1</span><span class='pln'> </span><span 
class='pun'>-</span><span class='pln'> tau</span><span 
class='pun'>)</span><span class='pln'> </span><span class='pun'>*</span><span 
class='pln'> apply</span><span class='pun'>(</span><span 
class='pln'>x</span><span class='pun'>,</span><span class='pln'> </span><span 
class='lit'>2</span><span class='pun'>,</span><span class='pln'> 
sum</span><span class='pun'>)</span><span class='pln'>
    d </span><span class='pun'><-</span><span class='pln'> rep</span><span 
class='pun'>(</span><span class='lit'>1</span><span class='pun'>,</span><span 
class='pln'> n</span><span class='pun'>)</span><span class='pln'>
    u </span><span class='pun'><-</span><span class='pln'> rep</span><span 
class='pun'>(</span><span class='lit'>1</span><span class='pun'>,</span><span 
class='pln'> n</span><span class='pun'>)</span><span class='pln'>
    wn </span><span class='pun'><-</span><span class='pln'> rep</span><span 
class='pun'>(</span><span class='lit'>0</span><span class='pun'>,</span><span 
class='pln'> </span><span class='lit'>10</span><span class='pln'> </span><span
 class='pun'>*</span><span class='pln'> n</span><span 
class='pun'>)</span><span class='pln'>
    wn</span><span class='pun'>[</span><span class='lit'>1</span><span 
class='pun'>:</span><span class='pln'>n</span><span class='pun'>]</span><span 
class='pln'> </span><span class='pun'><-</span><span class='pln'> </span><span
 class='pun'>(</span><span class='lit'>1</span><span class='pln'> </span><span
 class='pun'>-</span><span class='pln'> tau</span><span 
class='pun'>)</span><span class='pln'>
    z </span><span class='pun'><-</span><span class='pln'> </span><span 
class='pun'>.</span><span class='typ'>Fortran</span><span 
class='pun'>(</span><span class='str'>"rqfnb"</span><span 
class='pun'>,</span><span class='pln'> </span><span class='kwd'>as</span><span
 class='pun'>.</span><span class='pln'>integer</span><span 
class='pun'>(</span><span class='pln'>n</span><span class='pun'>),</span><span
 class='pln'> </span><span class='kwd'>as</span><span 
class='pun'>.</span><span class='pln'>integer</span><span 
class='pun'>(</span><span class='pln'>p</span><span class='pun'>),</span><span
 class='pln'> a </span><span class='pun'>=</span><span class='pln'> 
</span><span class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(</span><span 
class='pln'>t</span><span class='pun'>(</span><span class='kwd'>as</span><span
 class='pun'>.</span><span class='pln'>matrix</span><span 
class='pun'>(</span><span class='pln'>x</span><span 
class='pun'>))),</span><span class='pln'> 
        c </span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(-</span><span 
class='pln'>y</span><span class='pun'>),</span><span class='pln'> rhs 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(</span><span 
class='pln'>rhs</span><span class='pun'>),</span><span class='pln'> d 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(</span><span 
class='pln'>d</span><span class='pun'>),</span><span class='pln'> 
        </span><span class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(</span><span 
class='pln'>u</span><span class='pun'>),</span><span class='pln'> beta 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(</span><span 
class='pln'>beta</span><span class='pun'>),</span><span class='pln'> eps 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(</span><span 
class='pln'>eps</span><span class='pun'>),</span><span class='pln'> 
        wn </span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>as</span><span class='pun'>.</span><span 
class='kwd'>double</span><span class='pun'>(</span><span 
class='pln'>wn</span><span class='pun'>),</span><span class='pln'> wp 
</span><span class='pun'>=</span><span class='pln'> </span><span 
class='kwd'>double</span><span class='pun'>((</span><span class='pln'>p 
</span><span class='pun'>+</span><span class='pln'> </span><span 
class='lit'>3</span><span class='pun'>)</span><span class='pln'> </span><span 
class='pun'>*</span><span class='pln'> p</span><span 
class='pun'>),</span><span class='pln'> it</span><span 
class='pun'>.</span><span class='pln'>count </span><span 
class='pun'>=</span><span class='pln'> integer</span><span 
class='pun'>(</span><span class='lit'>3</span><span class='pun'>),</span><span
 class='pln'> 
        info </span><span class='pun'>=</span><span class='pln'> 
integer</span><span class='pun'>(</span><span class='lit'>1</span><span 
class='pun'>),</span><span class='pln'> PACKAGE </span><span 
class='pun'>=</span><span class='pln'> </span><span 
class='str'>"quantreg"</span><span class='pun'>)</span><span class='pln'>
    coefficients </span><span class='pun'><-</span><span class='pln'> 
</span><span class='pun'>-</span><span class='pln'>z$wp</span><span 
class='pun'>[</span><span class='lit'>1</span><span class='pun'>:</span><span 
class='pln'>p</span><span class='pun'>]</span><span class='pln'>
    names</span><span class='pun'>(</span><span 
class='pln'>coefficients</span><span class='pun'>)</span><span class='pln'> 
</span><span class='pun'><-</span><span class='pln'> dimnames</span><span 
class='pun'>(</span><span class='pln'>x</span><span 
class='pun'>)[[</span><span class='lit'>2</span><span 
class='pun'>]]</span><span class='pln'>
    residuals </span><span class='pun'><-</span><span class='pln'> y 
</span><span class='pun'>-</span><span class='pln'> x </span><span 
class='pun'>%*%</span><span class='pln'> coefficients
    list</span><span class='pun'>(</span><span class='pln'>coefficients 
</span><span class='pun'>=</span><span class='pln'> coefficients</span><span 
class='pun'>,</span><span class='pln'> tau </span><span 
class='pun'>=</span><span class='pln'> tau</span><span 
class='pun'>,</span><span class='pln'> residuals </span><span 
class='pun'>=</span><span class='pln'> residuals</span><span 
class='pun'>)</span><span class='pln'>
</span><span class='pun'>}</span></code>

For data vector of length 2000 i get:

(value = elapsed time in sec; columns = different number of columns of 
smoothed matrix/list)

<code><span class='pln'>           </span><span class='lit'>2cols</span><span 
class='pln'> </span><span class='lit'>4cols</span><span class='pln'> 
</span><span class='lit'>6cols</span><span class='pln'> </span><span 
class='lit'>8cols</span><span class='pln'>
apply      </span><span class='lit'>0.178</span><span class='pln'> 
</span><span class='lit'>0.096</span><span class='pln'> </span><span 
class='lit'>0.069</span><span class='pln'> </span><span 
class='lit'>0.056</span><span class='pln'>
lapply    </span><span class='lit'>16.555</span><span class='pln'> 
</span><span class='lit'>4.299</span><span class='pln'> </span><span 
class='lit'>1.785</span><span class='pln'> </span><span 
class='lit'>0.972</span><span class='pln'>
mc2lapply </span><span class='lit'>11.192</span><span class='pln'> 
</span><span class='lit'>2.089</span><span class='pln'> </span><span 
class='lit'>0.927</span><span class='pln'> </span><span 
class='lit'>0.545</span><span class='pln'>
mc4lapply </span><span class='lit'>10.649</span><span class='pln'> 
</span><span class='lit'>1.326</span><span class='pln'> </span><span 
class='lit'>0.694</span><span class='pln'> </span><span 
class='lit'>0.396</span><span class='pln'>
mc6lapply </span><span class='lit'>11.271</span><span class='pln'> 
</span><span class='lit'>1.384</span><span class='pln'> </span><span 
class='lit'>0.528</span><span class='pln'> </span><span 
class='lit'>0.320</span><span class='pln'>
mc8lapply </span><span class='lit'>10.133</span><span class='pln'> 
</span><span class='lit'>1.390</span><span class='pln'> </span><span 
class='lit'>0.560</span><span class='pln'> </span><span 
class='lit'>0.260</span></code>

For data of length 4000 i get:

<code><span class='pln'>            </span><span class='lit'>2cols</span><span
 class='pln'>  </span><span class='lit'>4cols</span><span class='pln'>  
</span><span class='lit'>6cols</span><span class='pln'> </span><span 
class='lit'>8cols</span><span class='pln'>
apply       </span><span class='lit'>0.351</span><span class='pln'>  
</span><span class='lit'>0.187</span><span class='pln'>  </span><span 
class='lit'>0.137</span><span class='pln'> </span><span 
class='lit'>0.110</span><span class='pln'>
lapply    </span><span class='lit'>189.339</span><span class='pln'> 
</span><span class='lit'>32.654</span><span class='pln'> </span><span 
class='lit'>14.544</span><span class='pln'> </span><span 
class='lit'>8.674</span><span class='pln'>
mc2lapply </span><span class='lit'>186.047</span><span class='pln'> 
</span><span class='lit'>20.791</span><span class='pln'>  </span><span 
class='lit'>7.261</span><span class='pln'> </span><span 
class='lit'>4.231</span><span class='pln'>
mc4lapply </span><span class='lit'>185.382</span><span class='pln'> 
</span><span class='lit'>30.286</span><span class='pln'>  </span><span 
class='lit'>5.767</span><span class='pln'> </span><span 
class='lit'>2.397</span><span class='pln'>
mc6lapply </span><span class='lit'>184.048</span><span class='pln'> 
</span><span class='lit'>30.170</span><span class='pln'>  </span><span 
class='lit'>8.059</span><span class='pln'> </span><span 
class='lit'>2.865</span><span class='pln'>
mc8lapply </span><span class='lit'>182.611</span><span class='pln'> 
</span><span class='lit'>37.617</span><span class='pln'>  </span><span 
class='lit'>7.408</span><span class='pln'> </span><span 
class='lit'>2.842</span></code>

Why is apply so much more efficient than mclapply? Maybe I'm just doing some
usual beginner mistake.

Thank you for your reactions.

	[[alternative HTML version deleted]]

