[R] Using R for Production - Discussion

[Saeed Abu Nimeh]

I worked on a project where we used a random forest classifier to
predict a binary response. We trained a model in the ec2 cloud with 3
million observations and 44 features. We stored the model that was
generated by R using save(mymodel,file="model.Rdata"). Now we use
model.Rdata locally to predict new observations.
In our local system, we built a parser in Perl to generate the csv
representation of the observation we want to predict, then we used
RSPerl to communicate between Perl and R. But there is a catch,
instead of loading the random forest model (model.Rdata) every time we
want to predict a new observation, we have an R console running as a
daemon with the model.Rdata loaded already. Then, we send the
observation to be predicted from Perl to R. If anyone else has better
solutions/ideas, please feel free to share.
Thanks,
Saeed

On Mon, Nov 1, 2010 at 9:04 PM, Santosh Srinivas
<santosh.srinivas at gmail.com> wrote:
> Hello Group,
>
> This is an open-ended question.
>
> Quite fascinated by the things I can do and the control I have on my
> activities since I started using R.
> I basically have been using this for analytical related work off my desktop.
> My experience has been quite good and most issues where I need to
> investigate and solve are typical items more related to data errors, format
> corruption, etc... not necessarily "R" Related.
>
> Complementing this with Python gives enough firepower to do lots of
> production (analytical related activities) on the cloud (from my research I
> see that every innovative technology provider seems to support Python ...
> google, amazon, etc).
>
> Question on using R for Production activities:
> Q1) Does anyone have experience of using R-scripts etc ... for production
> related activities. E.g. serving off a computational/ analytical /
> simulation environment from a webportal with the analytical processing done
> in R.
> I've seen that most useful things for normal (not rocket science) business
> (80-20 rule) can be done just as well in R in comparison with tools like
> SAS, Matlab, etc.
>
> Q2) I haven't tried the processing routines for much larger data-sets
> assuming "size" is not a constraint nowadays.
> I know that I should try out ... but any forewarnings would help. Is it
> likely that something that works for my "desktop" dataset is quite as likely
> to work when scaled up to a "cloud dataset"?
> Assuming that I do the clearing out of unused objects, not running into
> infinite loops, etc?
>
> i.e. is there any problem with the "fundamental architecture of R itself"?
> (like press articles often say)
>
>
> Q3) There are big fans of the SAS, Matlab, Mathworks environments out there
> .... does anyone have a comparison of how R fares.
> >From my experience R is quite neat and low level ... so overheads should be
> quite low.
> Most slowness comes due to lack of knowledge (see my code ... like using the
> wrong structures, functions, loops, etc.) rather than something wrong with
> the way R itself is.
> Perhaps there is no "commercial" focus to enhance performance related issues
> but my guess is that it is just matter of time till the community evolves
> the language to score higher on that too.
> And perhaps develops documentation to assist the challenge users with
> "performance tips" (the ten commandments types)
>
> Q4) You must have heard about the latest comment from James Goodnight of SAS
> ... "We haven't noticed that a lot. Most of our companies need industrial
> strength software that has been tested, put through every possible scenario
> or failure to make sure everything works correctly."
> My "gut" is that random passionate geeks (playing part-time) do better
> testing than a military of professionals ... (but I've no empirical evidence
> here)
>
> I am not taking a side here (although I appreciate those who do!) .. but
> looking for an objective reasoning.
>
> Thanks,
> S
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>