[Rd] Pre-compilation and server-side parallel execution

[Erik van Zijst]

Folks,

My company operates a platform that distributes real-time financial data 
from exchanges to users. To extend our services I want to allow users to 
write and submit custom R scripts to our platform that operate on our 
streaming data to do real-time analysis.

We have thousands of users deploying scripts and each script is 
evaluated repeatedly when certain conditions in the stream apply. For 
example, a script could compute the NASDAQ100 index value each time one 
of its 100 constituents trade.

Scripts are typically small and execute quickly. Each script is 
registered once and then repeatedly evaluated with different parameters 
(possibly several times per second per script). In this context my 
biggest concern is scalability.

The evaluation engine is a pure server-side component without display 
abilities. An R-script is invoked with parameters and whatever it 
returns is sent to the user.

Ideally I'd need a C api to interact with the interpreter. I've looked 
at projects like R/Apache, RServe and RSJava for inspiration and came to 
the conclusion that all these projects work by forking multiple 
instances of the R-engine where each instance evaluates one script at a 
time.

As our service must evaluate many different scripts concurrently 
(isolated from one another), I have the following concerns:

1. Spawning a pool of engine instances for massive parallel execution is 
expensive, but might work with lots of memory.
2. R's native C-api 
[http://cran.r-project.org/doc/manuals/R-exts.html#The-R-API] does not 
separate parsing from evaluation. When the same script is evaluated 10 
times, it is also parsed 10 times.

I'm mostly concerned about the second issue. Our scripts are registered 
once and continuously evaluated. I want to avoid parsing the same script 
again each time it is evaluated. Does the engine recognize previously 
parsed scripts (like oracle does for SQL queries)?

I interested to hear your thoughts on my concerns and whether you think 
R would work in this architecture.

kind regards,
Erik van Zijst
-- 
And on the seventh day, He exited from append mode.