[R] bam (mgcv) not using the specified number of cores

[Andrew Crane-Droesch]

Hi Simon,

(resending with all images as imgur so as to not bounce from list)

Thanks for the reply.  I've tried to reproduce the error, but I don't 
know how to show output from `top` any other way than with screenshots, 
so please excuse that.

Here are screenshots of what happens when I run with two
http://imgur.com/i26GKPo

and three
http://imgur.com/8SL7scy

cores.  In the former, it seems to be working on one core, and in the 
latter, it appears to be working on three.  When reproducing the error, 
I'm getting behavior that isn't entirely consistent -- sometimes it 
"behaves" and operates on the asked-for number of cores, and other times 
not.

I'm also attaching a screenshot
http://imgur.com/bJfuS6R
showing terminal output from a remote cluster when I run my full model 
(N=67K) rather than a subset (N=7K) -- I get that error "Error in 
qr.qty(qrx, f) : right-hand side should have 60650 not 118451 rows ..."  
I suppose this is a memory overload problem?  Any suggestions on how to 
get bam to not call for more memory than the node has available would be 
welcome, though I suspect that is a supercomputing problem rather than a 
mgcv problem.  I don't know much about memory management, except that R 
doesn't do it explicitly.

Thanks,
Andrew

sessionInfo() for local machine:
1> sessionInfo()
R version 3.0.2 (2013-09-25)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel  stats     graphics  grDevices utils datasets  methods
[8] base

other attached packages:
[1] mgcv_1.7-26  nlme_3.1-111

loaded via a namespace (and not attached):
[1] grid_3.0.2      lattice_0.20-23 Matrix_1.1-4
1>
On 08/21/2014 04:29 PM, Simon Wood wrote:
> Hi Andrew,
>
> Could you provide a bit more information, please. In particular the 
> results of sessionInfo() and the code that caused this weird behaviour 
> (+ an indication of dataset size).
>
> best,
> Simon
>
> On 21/08/14 12:53, Andrew Crane-Droesch wrote:
>> I am getting strange behavior when trying to fit models to large
>> datasets using bam.  I am working on a 4-core machine, but I think that
>> there may be 2 physical cores that the computer uses as 4 cores in some
>> sense that I don't understand.
>>
>> When I run the bam using makeCluster(3), the model runs on one core. But
>> when I run it on makeCluster(2), top shoes me that three of my cores are
>> taken up to full capacity, and my computer slows down or crashes.
>>
>> How can I get it to truly run on 2 cores?
>>
>> I'm on a thinkpad X230, running ubuntu.
>>
>> Thanks,
>> Andrew
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>