[R-sig-hpc] [zeromq-dev] rzmq package

Whit Armstrong armstrong.whit at gmail.com
Thu Sep 29 16:56:25 CEST 2011


The limits are determined by what can be represented by a call to 'serialize.'

zmq just cares about bytes across a wire, not whether they can be
assembled correctly on the other side.

So, for instance, database connections, external pointers, and other
such objects are not likely to be good cases for serialization.

However, normal vectors, dataframes, etc.  should all be fine.

If you want to send either some code to create a test object, or an
exported example object.  I can test on my site.

The cran package should be up shortly  (assuming I stomp out this last
Rd error).  Or you can always pull directly from github to test.

-Whit


On Thu, Sep 29, 2011 at 10:36 AM, Ulrich Staudinger
<ustaudinger at gmail.com> wrote:
> Awesome.
>
> Is there some limitation on the type of R structures that can be
> transmitted?
> I think about some list object that looks like this:
>
> jobResult = list()
> jobResult$jobId=123
> jobResult$status=FAILED
> jobResult$message="some exception text"
> jobResult$payload=NULL
>
> or:
> jobResult = list()
> jobResult$jobId=123
> jobResult$status=SUCCESS
> jobResult$payload=... some complex list object ...
>
> Would that be possible?
>
>
> Cheers,
> Ulrich
>
>
> Am 29.09.2011 15:45, schrieb Whit Armstrong:
>>
>> yes, indeed.  just need to hammer out the man pages for the R package.
>>  then I'll move on to the zmq bindings site.
>>
>> Cheers,
>> Whit
>>
>>
>> On Thu, Sep 29, 2011 at 9:29 AM, Martin Sustrik<sustrik at 250bpm.com>
>>  wrote:
>>
>>>
>>> Hi Whit,
>>>
>>> Nice! We've used R to process results from 0MQ benchmarking. Now we're
>>> back
>>> from a different direction.
>>>
>>> Would you mind creating a page on zero.mq website (it's a wiki) so that
>>> people know that the binding exists?
>>>
>>> Martin
>>>
>>> On 09/29/2011 05:03 AM, Daniel Cegiełka wrote:
>>>
>>>>
>>>> I will send your message to zmq-list!
>>>>
>>>> thx  Armstrong!
>>>> daniel
>>>>
>>>>
>>>> 2011/9/29 Whit Armstrong<armstrong.whit at gmail.com
>>>> <mailto:armstrong.whit at gmail.com>>
>>>>
>>>>    Just a quick post on a new package I've been wrapping up, rzmq.
>>>>
>>>>    https://github.com/armstrtw/rzmq
>>>>
>>>>    Finding inspiration in JD Long's segue package, and frustration with
>>>>    the config steps involved in dynamically updating the older debian
>>>>    distribution that Amazon uses for it's emr machines, I decided to try
>>>>    to hit the ec2 machines directly using my own ami.
>>>>
>>>>    The zmq messaging patterns allow one to distribute jobs across many
>>>>    nodes, but for now a simple example with only 1 micro instance.
>>>>
>>>>    the remote server:
>>>>
>>>>    ubuntu at ip-10-243-90-36:~$ cat remote.server2.r
>>>>    #!/usr/bin/env Rscript
>>>>
>>>>    library(rzmq)
>>>>    context = init.context()
>>>>    in.socket = init.socket(context,"ZMQ_PULL")
>>>>    bind.socket(in.socket,"tcp://*:5557")
>>>>
>>>>    out.socket = init.socket(context,"ZMQ_PUSH")
>>>>    bind.socket(out.socket,"tcp://*:5558")
>>>>
>>>>    while(1) {
>>>>        msg = receive.socket(in.socket);
>>>>        fun<- msg$fun
>>>>        args<- msg$args
>>>>        print(args)
>>>>        ans<- do.call(fun,args)
>>>>        send.socket(out.socket,ans);
>>>>    }
>>>>    ubuntu at ip-10-243-90-36:~$
>>>>
>>>>
>>>>    and the locally executed code:
>>>>
>>>>    estimatePi<- function(seed) {
>>>>        set.seed(seed)
>>>>        numDraws<- 1e5
>>>>
>>>>        r<- .5 #radius... in case the unit circle is too boring
>>>>        x<- runif(numDraws, min=-r, max=r)
>>>>        y<- runif(numDraws, min=-r, max=r)
>>>>        inCircle<- ifelse( (x^2 + y^2)^.5<  r , 1, 0)
>>>>
>>>>        sum(inCircle) / length(inCircle) * 4
>>>>    }
>>>>
>>>>
>>>>    print(system.time(ans<- zmq.lapply(as.list(1:1e2),
>>>>                                        estimatePi,
>>>>
>>>>
>>>>  execution.server="tcp://ec2-184-73-102-95.compute-1.amazonaws.com:5557
>>>>    <http://ec2-184-73-102-95.compute-1.amazonaws.com:5557>",
>>>>
>>>>    sink.server="tcp://ec2-184-73-102-95.compute-1.amazonaws.com:5558
>>>>    <http://ec2-184-73-102-95.compute-1.amazonaws.com:5558>")))
>>>>
>>>>    print(mean(unlist(ans)))
>>>>
>>>>    yields:
>>>>    warmstrong at krypton:~/dvl/zmq.test/test.ec2$ Rscript lapply.exmaple.r
>>>>       user  system elapsed
>>>>      0.010   0.010   7.007
>>>>    [1] 3.140964
>>>>
>>>>
>>>>    Anyway, I'll post up a better example tomorrow that actually uses
>>>> more
>>>>    than one machine.
>>>>
>>>>    -Whit
>>>>
>>>>    _______________________________________________
>>>>    R-sig-hpc mailing list
>>>>    R-sig-hpc at r-project.org<mailto:R-sig-hpc at r-project.org>
>>>>    https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>
>>>
>>
>> _______________________________________________
>> R-sig-hpc mailing list
>> R-sig-hpc at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>



More information about the R-sig-hpc mailing list