Hello,
I'll be releasing easy examples (and install scripts) for RHIPE on EC2 later
this week. I have tested it out on EC2 and it works smoothly, however I
haven't experimented with S3 datasets.

I'll send the link to the examples end of week.

Currently, I'm refactoring the code and it should be working in  a day (i
hope!).


Saptarshi


On Wed, May 13, 2009 at 4:40 PM, Avram Aelony <AvramAelony@eharmony.com>wrote:

> Hello,
>
> Is there documentation on how to use RHIPE with Amazon's EC2 using data
> from S3 ?
>
> Thank-you,
> Avram
>
>
>
>
> -----Original Message-----
> From: r-sig-hpc-bounces@r-project.org [mailto:
> r-sig-hpc-bounces@r-project.org] On Behalf Of Saptarshi Guha
> Sent: Saturday, May 02, 2009 12:01 PM
> To: malcolm Crouch
> Cc: R-sig-hpc@r-project.org
> Subject: Re: [R-sig-hpc] distributed R on EC2, designing the software stack
>
> Hello,
> I've been testing R across clusters and will execute some more in
> depth tests over the weekend.
> However, it depends on what Instance Types you use (types and cost:
> http://aws.amazon.com/ec2/instance-types/)
>
> I used a cluster of 15 "standard large instance", for less than  1 hr,
> which cost me $6. Intra EC2 bandwidth is free (in the same region),
> and downloading a GB from outside of the EC2 is ~$0.1 per GB. So, yes,
> depending on the budget (i'm a grad student and it is turning out to
> be expensive) it can easily run into hundreds of dollars on a monthly
> basis if used regularly.
>
> Keep in mind that if you have a web service running on an instance and
> check the website (e.g cluster status monitoring) from outside the EC2
> an expense is incurred. Also as soon as the instance is up and running
> you're being charged. Do not forget to terminate them!
>
> I'll be tabulating my performance and expenditure in the coming week.
>
> Hope it helps
> Saptarshi Guha
>
>
>
> On Sat, May 2, 2009 at 2:03 PM, malcolm Crouch
> <malcolm.croucher@gmail.com> wrote:
> >
> > Hi ,
> >
> >
> > i have looked at the pricing for Amazon , however am wondering from a
> > practical point of view what roughly your monthly costs are ?
> >
> > I am interested in using this service as I dont have access to a cluster
> but
> > need to do a feasilibilty study ... therefore some ideas on costing would
> be
> > useful.
> >
> > Regards
> >
> > Malcolm
> > On Wed, Apr 29, 2009 at 9:36 PM, Saptarshi Guha <
> saptarshi.guha@gmail.com>
> > wrote:
> >>
> >> Hello,
> >> Yes, I was playing with EC2 and Rhipe last night. Just got permission
> >> to increase my instances to 100!
> >> The details (what I know)
> >> RHIPE is based on Hadoop and R. Cloudera has a very easy to use AMI
> >> for small and large (32/64bit) instances. It is easy enough to install
> >> the cloudera AMI.
> >> However It does not come with R.
> >>
> >> Last night, I modified their scripts to yum install R (using yum we
> >> get R-2.6) on each machine - as such this results in ~21MB downloads
> >> on the machines[1], which is not expensive but is not the best way do
> >> things.
> >>
> >> Once booted, each machine installs R, Rserve and one machine (the
> >> master) installs RHIPE.
> >> I did it with 1 master and 1 tasktracker and RHIPE worked. I intend to
> >> check with 30+ instances to see how things scale.
> >>
> >> I have emailed cloudera asking them to bundle R with their Hadoop AMI
> >> - so that users incur a minimal expense.
> >>
> >> I will be placing EC2 instructions to use RHIPE shortly this week.
> >> Given the reasonable cost of EC2, it would be a great way for users to
> >> test out distributed computing with R. Maybe as part of the R
> >> community we could host a linux AMI? Again, cost is the issue
> >> here(rather not pay for users downloading things)
> >>
> >> Regards
> >> Saptarshi Guha
> >> [1] Not quite sure how the AMI's work - if 10 AMIs belong to one
> >> group, does EC2 boot up one and replicate the booted instance? If so,
> >> then there is only one download, if not each machine downloads.
> >>
> >>
> >> On Wed, Apr 29, 2009 at 3:24 PM, Whit Armstrong
> >> <armstrong.whit@gmail.com> wrote:
> >> > you should contact Robert Grossman who just gave a presentation on
> >> > this topic at R/Finance in Chicago.
> >> >
> >> > link: http://rinfinance.quantmod.com/speakers/
> >> >
> >> > -Whit
> >> >
> >> >
> >> > On Wed, Apr 29, 2009 at 3:06 PM, Stephen J. Barr
> >> > <stephenjbarr@gmail.com> wrote:
> >> >> Greetings,
> >> >>
> >> >> I am trying to get into distributed computing with R, but do not have
> >> >> access to a cluster. Therefore, I am trying to get distributed R
> >> >> running on Amazon's EC2. ( http://aws.amazon.com/ec2/ )
> >> >>
> >> >> For those of you who don't know, EC2 allows you to instantiate large
> >> >> numbers of computers, bundled with whatever OS and software
> >> >> configuration you want. From my survey of things, there are a lot of
> >> >> different options available for distributed computing. For my needs,
> I
> >> >> would just like to run simple Monte Carlo simulations, and other
> >> >> things that don't require a ton of inter-node communication.
> >> >>
> >> >> What I would like to do is put together a public AMI and a howto
> >> >> guide, such that it would be very easy for anyone to instantiate an
> >> >> N-node cluster and start with parallel computing. I would like to
> have
> >> >> a discussion/brainstorm over what the exact software stack should be.
> >> >>
> >> >> My initial thoughts were:
> >> >>
> >> >> 1) R 2.9.0 + OpenMPI + RMpi + Snowfall/sfCluster
> >> >>   - will Amazon's network work with OpenMPI. Perhaps it would be
> >> >> better to use PVM or something that is more tolerant to non-optimal
> >> >> network
> >> >>
> >> >> 2)  R 2.9.0 + "socket based communication" + Snowfall/sfCluster
> >> >>  - is this scalable
> >> >>
> >> >> 3)  R 2.9.0 + twisted + NetWorkSpaces
> >> >>   - not sure of Amazon's network supports broadcast mode, which is
> >> >> required by twisted
> >> >>
> >> >> 4) Biocep-R
> >> >>   - this looks like it has the functionality to do what I want, but a
> >> >> lot of other stuff as well.
> >> >>
> >> >> 5) RHIPE
> >> >>   - Hadoop is well supported by EC2. Perhaps this is the way to go.
> >> >> Seems like a very new package :)
> >> >>
> >> >> What are people's thoughts on what would be a good software stack
> with
> >> >> the constraint that it should be simple and run on EC2?
> >> >>
> >> >> Thanks,
> >> >> -stephen
> >> >> ==========================================
> >> >> Stephen J. Barr
> >> >> University of Washington
> >> >> WEB: www.econsteve.com
> >> >>
> >> >> _______________________________________________
> >> >> R-sig-hpc mailing list
> >> >> R-sig-hpc@r-project.org
> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >> >>
> >> >
> >> > _______________________________________________
> >> > R-sig-hpc mailing list
> >> > R-sig-hpc@r-project.org
> >> > https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >> >
> >>
> >> _______________________________________________
> >> R-sig-hpc mailing list
> >> R-sig-hpc@r-project.org
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >
> >
> >
> > --
> > Malcolm A.B Croucher
> >
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>

	[[alternative HTML version deleted]]

