[R] bam (mgcv) not using the specified number of cores

[Andrew Crane-Droesch]

Hi Simon,

Thanks for the reply.  I've tried to reproduce the error, but I don't 
know how to show output from `top` any other way than to attach 
screenshots, so please excuse that.

I'm attaching screenshots of what happens when I run with two and three 
cores.  In the former, it seems to be working on one core, and in the 
latter, it appears to be working on three.  When reproducing the error, 
I'm getting odd behavior that isn't entirely consistent -- sometimes it 
"behaves" and operates on the asked-for number of cores, and other times 
not.

I'm also attaching a screenshot showing terminal output from a remote 
cluster when I run my full model (N=67K) rather than a subset (N=7K) -- 
I get that error "Error in qr.qty(qrx, f) : right-hand side should have 
60650 not 118451 rows ..."  I suppose this is a memory overload 
problem?  Any suggestions on how to get bam to not call for more memory 
than the node has available would be welcome, though I suspect that is a 
supercomputing problem rather than a R problem.

Thanks,
Andrew

sessionInfo() for local machine:
1> sessionInfo()
R version 3.0.2 (2013-09-25)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel  stats     graphics  grDevices utils datasets  methods
[8] base

other attached packages:
[1] mgcv_1.7-26  nlme_3.1-111

loaded via a namespace (and not attached):
[1] grid_3.0.2      lattice_0.20-23 Matrix_1.1-4
1>





On 08/21/2014 04:29 PM, Simon Wood wrote:
> Hi Andrew,
>
> Could you provide a bit more information, please. In particular the 
> results of sessionInfo() and the code that caused this weird behaviour 
> (+ an indication of dataset size).
>
> best,
> Simon
>
> On 21/08/14 12:53, Andrew Crane-Droesch wrote:
>> I am getting strange behavior when trying to fit models to large
>> datasets using bam.  I am working on a 4-core machine, but I think that
>> there may be 2 physical cores that the computer uses as 4 cores in some
>> sense that I don't understand.
>>
>> When I run the bam using makeCluster(3), the model runs on one core. But
>> when I run it on makeCluster(2), top shoes me that three of my cores are
>> taken up to full capacity, and my computer slows down or crashes.
>>
>> How can I get it to truly run on 2 cores?
>>
>> I'm on a thinkpad X230, running ubuntu.
>>
>> Thanks,
>> Andrew
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2014-08-21 18:33:36.png
Type: image/png
Size: 576802 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20140821/41be1e87/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2014-08-21 18:34:42.png
Type: image/png
Size: 387719 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20140821/41be1e87/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2014-08-21 18:36:08.png
Type: image/png
Size: 388816 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20140821/41be1e87/attachment-0005.png>