[R] Problem with memory consuming algorithm

[jim holtman]

The first thing I would do, especially before letting it run for 48
hours, is to take a small subset of the data and run it with the
profiler (Rprof) enable to see where time is being spent.  I would
also put some print statements in the main loop to periodically output
the amount of CPU and memory being used.  It is hard to look through
the code without a subset of the data.  So run the profiler and tell
us what the output looks like.  You may be calling a function that is
not part of your code that is taking all the time, but until you run
the profiler, it is hard to tell.

On Fri, Sep 16, 2011 at 4:45 PM, Simon Zehnder <szehnder at uni-bonn.de> wrote:
> Hi guyz,
>
> I have serious problems with an algorithm I let run on a supercomputer:
>
> You find the functions under the following URLs:
>
> simuFunctionCaller: http://pastebin.com/6gw2fJFb
>
> calls Function simuFunctionNBM (http://pastebin.com/QeJDUnqx)
>
> after reading a csv-file ordered like the following:
>
> ,name,theta,std_error_theta,theta_phi,std_error_theta_phi,phi,N,sigma_epsilon,sigma_eta
> 31,25%,0.003044,2.28E-09,0.00742575,4.71E-09,0.00452525,68164.75,0.008706072,0.00774016
> 32,50%,0.004536,3.22E-09,0.01045,7.94E-09,0.0052762,110150,0.012387386,0.011169953
> 33,75%,0.007356625,1.24E-08,0.014249075,3.16E-08,0.007518,178840.5,0.018471461,0.016058424
>
> Reading and running without an exception is no problem. But the algorithm needs hours to run and seems to use more memory than available:
>
> I have given a time limit of 48 h and a memory limit of 128 GB RAM on the supercomputer (it is a cluster, a BATCH System)…..nevertheless the computer shuts down the algorithm after a certain time because of memory problems.
>
> Does anyone of u guys see a problem in the algorithm? Especially with so much RAM and time available?
>
> I am very thankful for your suggestions!
>
> Simon
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



-- 
Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?