[R-SIG-Mac] R run time question

[Simon Urbanek]

On Jul 13, 2011, at 3:50 PM, Steven McKinney wrote:

> 
> 
> 
>> -----Original Message-----
>> From: r-sig-mac-bounces at r-project.org [mailto:r-sig-mac-bounces at r-project.org] On Behalf Of Wayne Gray
>> Sent: July-13-11 12:25 PM
>> To: r-sig-mac at r-project.org
>> Subject: [R-SIG-Mac] R run time question
>> 
>> All,
>> 
>> We are doing ANOVAs that take a long time (> 3 hrs) to run in 64-bit mode on an Intel MacPro with 8
>> gbytes and 2.66 GHZ Intel Core i7 machine.
>> 
>> We have just tried running these on our server where we don't care as much how long it takes. The
>> server is a Mac "2 x 3 GHz Dual-Core Intel Xeon" with 13 GB 667 MHz DDR2 FB-DIMM.
>> 
>> We have installed R "R 2.10.1 GUI 1.31-np Tiger build 32-bit (5538)" on the server.
> 
> Why are you installing a 32 bit R?  And why an older version of R?
> 
> Can you not install a 64 bit version of the latest R?
> 
>> 
>> The data.frame is composed of the following observations, factors, and numerals:
>> 
>>> str(e1jit41blkAll)
>> 'data.frame':	21648 obs. of  6 variables:
>> $ subjectid: Factor w/ 22 levels "1379","1744",..: 1 1 1 1 1 1 1 1 1 1 ...
>> $ cond     : Factor w/ 2 levels "Visual","Auditory": 1 1 1 1 1 1 1 1 1 1 ...
>> $ block    : Factor w/ 8 levels "2","3","4","5",..: 1 1 1 1 1 1 1 1 1 1 ...
>> $ cbtime   : Factor w/ 41 levels "1","2","3","4",..: 1 10 11 12 13 14 15 16 17 18 ...
>> $ dirtime  : Ord.factor w/ 3 levels "Early"<"Middle"<..: 1 1 1 1 1 1 1 1 1 1 ...
>> $ jitter   : num  7.78 7.33 5.56 5 6.11 ...
>> 
>> When we run the following ANOVA on this dataframe:
>> 
>>> anova.ALL.e1jit41.blkAll  <- with(e1jit41blkAll, aov(jitter ~ cond*block*dirtime*cbtime +
>> error(subjectid/(block*dirtime*cbtime)), data = e1jit41blkAll))
>> 
>> We get the following feedback:
>> 
>> Error: cannot allocate vector of size 3.5 Gb
> 
> Running the ANOVA requires more than 4GB of RAM, which can not be done
> with 32 bit versions of software.  So you will need 64 bit R to do this analysis.
> 
>> 
>>> 
>> R(29720,0xa000d000) malloc: *** vm_allocate(size=3749089280) failed (error code=3)
>> R(29720,0xa000d000) malloc: *** error: can't allocate region
>> R(29720,0xa000d000) malloc: *** set a breakpoint in szone_error to debug
>> R(29720,0xa000d000) malloc: *** vm_allocate(size=3749089280) failed (error code=3)
>> R(29720,0xa000d000) malloc: *** error: can't allocate region
>> R(29720,0xa000d000) malloc: *** set a breakpoint in szone_error to debug
>>> 
>> 
>> So two categories of questions for all of you Mac-R wizards out there.
>> 
>> First category of question: can this analysis be run faster (and if so, how) on our laptop Intel Macs?
>> 
>> Second category of question: is there anything that can be done to the Apple Server or to our analysis
>> so that the analysis runs on our server?
> 
> What OS version is on your server?  If it's an old 32 bit version, upgrading the server
> to a recent 64 bit OS will help.  Then install a recent 64 bit version of R.
> 

FWIW: you can run R in 64-bit on Tiger as well, you just have to disable Quartz and the GUI since Tiger doesn't have 64-bit Cocoa.

Cheers,
Simon



> If you can do this, you can better assess whether this analysis can be done 
> faster on the 8GB laptops.
> 
> R runs entirely in RAM, so if the analysis needs more than 8GB of RAM, you may
> be experiencing "swapping" on the laptops.  The operating system allocates
> as much memory as the job needs - if that amount of memory exceeds the amount
> of RAM you have, then the rest is allocated to "virtual memory" which is
> on the disk drive.  Swapping occurs when part of the virtual memory of your job
> that is on disk is needed - the OS has to copy a chunk of what is in RAM out to
> disk, so it can copy the other chunk on disk back into RAM.
> 
> Monitor the laptop with a shell command line process such as "top" 
> and you will be able to determine how much virtual memory your R job
> is using, and how many "swaps" are involved.  Swapping really slows things
> down as reading and writing from the disk is slow.  If your job is swapping
> on the laptop, but uses less than 13GB of memory in total, it will run faster
> on the server.  If the job is not swapping at all on the laptop,  it will not
> run appreciably faster on the server.
> 
> 
> HTH
> 
> Steve McKinney
> 
> 
>> 
>> Many thanks,
>> 
>> Wayne Gray
>> 
>> 	[[alternative HTML version deleted]]
>> 
>> _______________________________________________
>> R-SIG-Mac mailing list
>> R-SIG-Mac at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> 
> _______________________________________________
> R-SIG-Mac mailing list
> R-SIG-Mac at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> 
>