[R-SIG-Mac] single-threaded R, 100% CPU with BLAS, vecLib and ATLAS
Melanie Courtot
mcourtot at gmail.com
Fri Oct 26 19:18:34 CEST 2012
Hi Ray and Simon, all,
Thanks for the help. My laptop has 8GB of RAM (my colleague has 12 on his desktop). I ssh'ed into his machine and the whole file loads in not even 2 seconds.
The file is read with mat<-read.csv('test.arff',header=FALSE,sep=',') The arff file is what I use with Weka, which is basically a comma delimited file. It contains around 7.5M datapoints (6200 rows, 1140 columns)
It seems that with 8GB I should be quite ok?
Based on your suggestions I tried with a part of the file only, which does work fine, so it seems that it is indeed a memory problem. Any idea as to why?
Thanks,
Melanie
Example record (I have 6200 of those)
856243,negative,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
On 2012-10-25, at 6:39 PM, Ray DiGiacomo, Jr. wrote:
> Hi Simon,
>
> I took the spec from this Revo SlideShare. The spec is based on a regression.
>
> http://www.revolutionanalytics.com/news-events/free-webinars/2011/intro-to-r-for-sas-spss/
>
> Click the right arrow until you get to slide 3 of 14. Then, look at the slide in the lower-right hand corner (slide 12).
>
> - Ray
>
>
>
>
>
> On Thu, Oct 25, 2012 at 6:26 PM, Simon Urbanek <simon.urbanek at r-project.org> wrote:
>
> On Oct 25, 2012, at 7:42 PM, Ray DiGiacomo, Jr. wrote:
>
> > Hello Melanie,
> >
> > How much RAM is installed on your MacBook Pro compared to your colleague's
> > Linux machine?
> >
> > How big is your dataset in terms of rows and columns?
> >
> > I believe R can handle about 10M datapoints per GB of RAM.
> >
>
> What exactly is that an estimate of? In R, 1GB of RAM will store ~134Mio datapoints when using numeric matrices/vectors and twice as many as integers or logicals. In practice, you will still need some room for computation on the data, though.
>
> Cheers,
> Simon
>
>
> > Note that datapoints = rows x columns
> >
> > Best Regards,
> >
> > Ray DiGiacomo, Jr.
> > Master R Trainer
> > President, Lion Data Systems LLC
> > President, The Orange County R User Group
> > Board Member, TDWI
> > rayd at liondatasystems.com
> > (Mobile) 408-425-7851
> > San Juan Capistrano, California
> >
> > Check out my one-on-one web-based R courses at liondatasystems.com/courses
> >
> >
> >
> >
> >
> > On Thu, Oct 25, 2012 at 4:16 PM, Melanie Courtot <mcourtot at gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I am trying to run R on my MacBook Pro 2.4 GHz Intel core i5. I am trying
> >> to read a csv file, which works fine on my work colleague's machine (under
> >> linux) but causes my CPU to go up to 100% and makes the GUI unresponsive
> >> and hangs on the command line. Activity monitor indicates there is only one
> >> R thread running.
> >>
> >> I did see that by default R was using the BLAS library, which is
> >> single-threaded, and that there was an option to use vecLib instead. I did
> >> this, and
> >> ls -l /Library/Frameworks/R.framework/Resources/lib/libRblas.dylib
> >> does return
> >> /Library/Frameworks/R.framework/Resources/lib/libRblas.dylib ->
> >> libRblas.vecLib.dylib
> >>
> >> I however still see the same behavior: 100% CPU, single thread.
> >>
> >> I saw that some MacBook pro (Xeon Nehalem based) had a vecLib bug, so I
> >> built the ATLAS library and symlinked R to libtatlas.dylib (unfortunately
> >> the pre compiled binairies pointed to in a previous email on the list [1]
> >> were not available anymore. Building ATLAS was... fun ;)) I was able to get
> >> the shared libraries (using --shared in my config) but still see the same
> >> behavior when trying to run my code. I was unsure if I should link to
> >> libsatlas.dylib or libtatlas.dylib, so tried both (I guess the latter was
> >> the right one though)
> >>
> >> I tried building R from the source (specifying -arch x86_64 and
> >> --enable-BLAS-shlib to be able to switch libraries), but same behavior and
> >> it seems it is an identical version to the prepackaged one (I tried with
> >> BLAS, vecLib and ATLAS)
> >>
> >> R info: R version 2.15.1 (2012-06-22) -- "Roasted Marshmallows", Platform:
> >> x86_64-apple-darwin9.8.0/x86_64 (64-bit)
> >>
> >> Any help would be greatly appreciated.
> >>
> >> Thanks,
> >> Melanie
> >>
> >>
> >> [1] https://stat.ethz.ch/pipermail/r-sig-mac/2010-October/007817.html
> >>
> >> ---
> >> Mélanie Courtot
> >> MSFHR/PCIRN Ph.D. Candidate,
> >> BCCRC - Terry Fox Laboratory - 12th floor
> >> 675 West 10th Avenue
> >> Vancouver, BC
> >> V5Z 1L3, Canada
> >>
> >> _______________________________________________
> >> R-SIG-Mac mailing list
> >> R-SIG-Mac at r-project.org
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> >>
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-SIG-Mac mailing list
> > R-SIG-Mac at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>
>
More information about the R-SIG-Mac
mailing list