Thanks Glenn. Your help is much appreciated - I will talk to my admin folks
here.

Srihari


On Tue, Oct 22, 2013 at 1:46 PM, Lockwood, Glenn <glock@sdsc.edu> wrote:

> Srihari,
>
> mpd is a part of mpich (and mvapich/intel mpi), not OpenMPI, so your
> application probably isn't running the way you anticipated at all.  This is
> probably due to your login environment ($PATH, $LD_LIBRARY_PATH, etc) not
> being set up to use your new OpenMPI installation.
>
> The errors you sent in the previous post sound like system-specific issues
> that require information specific to your cluster--it may be more effective
> to work with your local cluster admin (if you have one!) to iron out the
> issues arising from your MPI stack.  I don't think the problem is with R,
> but just take a step back and make sure you have the necessary essentials:
>
> 1. Your MPI stack needs to be compiled with a specific compiler (it sounds
> like you have Intel and GCC)
> 2. Your R needs to also be compiled with that same compiler
> 3. Your Rmpi needs to be compiled against the MPI stack in #1 and the same
> compiler as #1 and #2 (this may be non-trivial)
> 4. All of these need to be accessible from your local environment (either
> via some "module" command, which is common on most clusters, or by
> correctly setting $PATH, $LD_LIBRARY_PATH, etc in your .bashrc or .cshrc)
>
> If you are missing (or unsure) of any of these points, your local cluster
> admin is probably best qualified to sort these out for you.  Good luck!
>
> Glenn
>
>
> On Oct 22, 2013, at 10:05 AM, Srihari Radhakrishnan <srihari@iastate.edu>
> wrote:
>
> > Update:
> >
> > I just installed a local version of Open MPI (1.6.5) and reran the job.
> > Here's the output from the log -
> >
> >> source("http://bioconductor.org/biocLite.R")
> >> #MPI stuff initialization
> >> library(Rmpi)
> >> library(foreach)
> >> library(doMPI)
> >> cl <- startMPIcluster(count=31)
> > mpiexec_node228: cannot connect to local mpd (/tmp/mpd2.console_srihari);
> > possible cau
> > ses:
> >  1. no mpd is running on this host
> >  2. an mpd is running but was started without a "console" (-n option)
> >
> > I did not realize I had to start mpdboot before running mpirun - so I
> > sandwiched the mpirun command with:
> >
> > mpdboot -f ${PBS_NODEFILE}  -n 2
> > time mpirun -n 1 R --slave -f ParallelAnalysis.R
> > mpdallexit
> >
> > When I do that, here's what I run into:
> >
> > [unset]: Command cmd=put kvsname=singinit_kvs_3676_0
> key=P0-businesscard-1
> > value=rdma_
> >
> port0#3676$rdma_host0#2:0:0:172:16:0:228:0:0:0:0:0:0:0:0$fabrics_list#shm_and_dapl$
> > failed, reason='"'singinit_kvs_3676_0'"'
> > Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves =
> count,
> > :
> >  Other MPI error, error stack:
> > MPI_Comm_spawn(170).............:
> MPI_Comm_spawn(cmd="/work/srihari/program
> > s/R-3.0.1/lib64/R/bin/Rscript", argv=0x2982680, maxprocs=31,
> MPI_INFO_NULL,
> > root=0, MP
> > I_COMM_SELF, intercomm=0x25c03f0, errors=0x2e25bd0) failed
> > MPIDI_Comm_spawn_multiple(123)..:
> > MPIDI_CH3_Dynamic_processes(145):
> > MPID_nem_reinit(1363)...........:
> > MPIDI_PG_SetConnInfo_async(780).: PMI_KVS_Put returned -1
> > Calls: startMPIcluster -> mpi.comm.spawn -> .Call
> > Execution halted
> >
> --------------------------------------------------------------------------
> > mpirun noticed that the job aborted, but has no info as to the process
> > that caused that situation.
> >
> --------------------------------------------------------------------------
> >
> > Pretty lost at the moment - any pointers would greatly help me!
> >
> > Thanks
> > Srihari
> >
> > On Tue, Oct 22, 2013 at 10:13 AM, Srihari Radhakrishnan <
> srihari@iastate.edu
> >> wrote:
> >
> >> Thanks, Glenn and Steve for your input!
> >>
> >> As you suggested, I switched to the Open MPI implementation of mpirun -
> >> but I get another error this time:
> >>
> >> *[node205:02122] [[63174,1],3] ORTE_ERROR_LOG: A message is attempting
> to
> >> be sent to a*
> >> *process whose contact information is unknown in file rml_oob_send.c at
> >> line 104*
> >> *[node205:02122] [[63174,1],3] could not get route to
> [[INVALID],INVALID]*
> >>
> >> A little bit of googling told me to set my LD_LIBRARY_PATH variable to
> >> include the openmpi libs - which I added to my ~/.bashrc as follows:
> >>
> >> LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/shared/openmpi-1.6.4/gcc-4.4.6/lib/
> >> export LD_LIBRARY_PATH
> >>
> >> I also sourced my ~/.bashrc file in the first line of my PBS script so
> the
> >> LD_LIBRARY_PATH is reflected in all of the spawned workers as well. No
> luck
> >> still - I still get the same error listed above.
> >>
> >> Steve, I also tried running mpirun without the "-n 1" parameter - but
> like
> >> you said, it spawned off multiple master processes, just like Glenn said
> >> would happen. Running startMPIcluster() without the count=31 argument
> >> didn't change the respective errors with either compiler
> (intel/openmpi).
> >>
> >> Now, I am not sure how the cluster admin folks compiled openmpi - if
> they
> >> did it with intel compilers or gcc is not known to me at the moment.
> But if
> >> they compiled openmpi with intel compilers, could that be part of the
> >> problem?
> >>
> >> I sincerely appreciate all the help!
> >>
> >>
> >> On Tue, Oct 22, 2013 at 5:00 AM, <r-sig-hpc-request@r-project.org>
> wrote:
> >>
> >>> Send R-sig-hpc mailing list submissions to
> >>>        r-sig-hpc@r-project.org
> >>>
> >>> To subscribe or unsubscribe via the World Wide Web, visit
> >>>        https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >>> or, via email, send a message with subject or body 'help' to
> >>>        r-sig-hpc-request@r-project.org
> >>>
> >>> You can reach the person managing the list at
> >>>        r-sig-hpc-owner@r-project.org
> >>>
> >>> When replying, please edit your Subject line so it is more specific
> >>> than "Re: Contents of R-sig-hpc digest..."
> >>>
> >>>
> >>> Today's Topics:
> >>>
> >>>   1. Help with doMPI on multiple cores on a cluster
> >>>      (Srihari Radhakrishnan)
> >>>   2. Re: Help with doMPI on multiple cores on a cluster
> >>>      (Stephen Weston)
> >>>   3. Re: Help with doMPI on multiple cores on a cluster
> >>>      (Lockwood, Glenn)
> >>>
> >>>
> >>> ----------------------------------------------------------------------
> >>>
> >>> Message: 1
> >>> Date: Mon, 21 Oct 2013 08:42:35 -0500
> >>> From: Srihari Radhakrishnan <srihari@iastate.edu>
> >>> To: r-sig-hpc@r-project.org
> >>> Subject: [R-sig-hpc] Help with doMPI on multiple cores on a cluster
> >>> Message-ID:
> >>>        <CACq2iiM0NRo=
> >>> 12JGzquezyrHd7WyDcL9K_O+uSg50itpJ8f9OQ@mail.gmail.com>
> >>> Content-Type: text/plain
> >>>
> >>> Hi,
> >>>
> >>> I've been trying to use the doMPI to run the iterations of a for loop
> in
> >>> parallel (using the foreach package) on a cluster. However, I've been
> >>> running into issues - I think its the way I am running the R script,
> but I
> >>> could be wrong. Here's the description of the problem.
> >>>
> >>> We use a PBS scheduler to submit jobs; my script uses 2 nodes (32
> cores)
> >>> for now. I run 1 version of the R interpreter which internally calls 31
> >>> workers using R's mpi libraries. I produce below the PBS script, the R
> >>> code
> >>> (the relevant bits) and the error.
> >>>
> >>> ***Begin PBS Script***
> >>> #!/bin/bash
> >>>
> >>> #PBS  -o BATCH_OUTPUT
> >>> #PBS  -e BATCH_ERRORS
> >>>
> >>> #PBS -lnodes=2:ppn=16:compute,walltime=12:00:00
> >>>
> >>> # Change to directory from which qsub command was issued
> >>>   cd $PBS_O_WORKDIR
> >>>
> >>> cat $PBS_NODEFILE
> >>> #Call mpirun with 1 copy of the R interpreter. This will spawn 31
> workers,
> >>> inside the R script
> >>> time mpirun -n 1 R --slave -f ParallelAnalysis.R
> >>> ***End PBS script***
> >>>
> >>> ***Begin R Script***
> >>> source("http://bioconductor.org/biocLite.R")
> >>> #MPI stuff initialization
> >>> library(Rmpi)
> >>> library(foreach)
> >>> library(doMPI)
> >>> cl <- startMPIcluster(count=31) #call 31 clusterworkers/slaves
> >>> registerDoMPI(cl)
> >>> library(MEDIPS)
> >>> library(BSgenome)
> >>> .
> >>> .
> >>> *more R code; variable assignments etc; no mpi stuff here*
> >>> .
> >>> .
> >>> #Following code will run 100 parallel iterations using the doMPI
> library
> >>> loaded above and output results to the variable x. x is a table and
> stores
> >>> results from iterations as rows.
> >>> x <-foreach(i=1:100,.combine='rbind') %dopar% {
> >>> *stuff to do inside loop*
> >>> }
> >>> write.table(x, "output.tsv") #write x into file.
> >>> ***End R script***
> >>>
> >>> The execution halts as soon as the libraries are loaded - I get the
> >>> following error message repeatedly from both nodes (node 203 and node
> 202)
> >>>
> >>> *[2:node203] unexpected disconnect completion event from dynamic
> process
> >>> with rank=0 pg*
> >>> *_id=kvs_17890_0 0x1fce600*
> >>> *Assertion failed in file ../../dapl_conn_rc.c at line 1128: 0*
> >>>
> >>> I am not sure if this is an issue with the compilers or the script
> itself.
> >>> The script runs successfully without using mpi (using only 1 node). Any
> >>> help would be highly appreciated.
> >>>
> >>> Thanks in advance,
> >>> Srihari
> >>>
> >>> --
> >>> Srihari Radhakrishnan
> >>>
> >>> Ph.D candidate
> >>> Valenzuela Lab
> >>> Iowa State University
> >>>
> >>>        [[alternative HTML version deleted]]
> >>>
> >>>
> >>>
> >>> ------------------------------
> >>>
> >>> Message: 2
> >>> Date: Mon, 21 Oct 2013 09:55:55 -0400
> >>> From: Stephen Weston <stephen.b.weston@gmail.com>
> >>> To: Srihari Radhakrishnan <srihari@iastate.edu>
> >>> Cc: "R-sig-hpc@r-project.org" <r-sig-hpc@r-project.org>
> >>> Subject: Re: [R-sig-hpc] Help with doMPI on multiple cores on a
> >>>        cluster
> >>> Message-ID:
> >>>        <CALh21iK7Zap9UYstvxYm7sjBLX0=
> >>> P3KzZUE9OLkDzfdkQjg89w@mail.gmail.com>
> >>> Content-Type: text/plain; charset=ISO-8859-1
> >>>
> >>> Hi Srihari,
> >>>
> >>> I suspect it's an MPI issue.  Are you able to run any other simple MPI
> >>> programs successfully, and specifically, any using R with Rmpi?  From
> >>> the error message, it appears that you're using Intel MPI, which I've
> >>> never used.  I believe Rmpi is primarily tested with Open MPI, which
> >>> is what I've always used with doMPI.  It would be interesting to see
> >>> if you can run successfully using Open MPI, if that is possible for
> >>> you.
> >>>
> >>> You'll probably need to look for help on an Intel MPI forum, although
> >>> you may need to reduce the problem to something that doesn't use R.
> >>>
> >>> Here is a similar issue that I found on an Intel MPI forum:
> >>>
> >>>    http://software.intel.com/en-us/forums/topic/329053
> >>>
> >>> You could also try running without spawning, since that may be a
> >>> problem for Intel MPI.  To do that, change the R script to use:
> >>>
> >>>    cl <- startMPIcluster()
> >>>
> >>> Also change the mpirun command in the PBS script to use '-n 32' or
> >>> don't specify the -n option at all.  In that case, mpirun will start
> >>> all of the workers as well as the master which may work better.
> >>>
> >>> Regards,
> >>>
> >>> Steve Weston
> >>>
> >>> On Mon, Oct 21, 2013 at 9:42 AM, Srihari Radhakrishnan
> >>> <srihari@iastate.edu> wrote:
> >>>> Hi,
> >>>>
> >>>> I've been trying to use the doMPI to run the iterations of a for loop
> in
> >>>> parallel (using the foreach package) on a cluster. However, I've been
> >>>> running into issues - I think its the way I am running the R script,
> >>> but I
> >>>> could be wrong. Here's the description of the problem.
> >>>>
> >>>> We use a PBS scheduler to submit jobs; my script uses 2 nodes (32
> cores)
> >>>> for now. I run 1 version of the R interpreter which internally calls
> 31
> >>>> workers using R's mpi libraries. I produce below the PBS script, the R
> >>> code
> >>>> (the relevant bits) and the error.
> >>>>
> >>>> ***Begin PBS Script***
> >>>> #!/bin/bash
> >>>>
> >>>> #PBS  -o BATCH_OUTPUT
> >>>> #PBS  -e BATCH_ERRORS
> >>>>
> >>>> #PBS -lnodes=2:ppn=16:compute,walltime=12:00:00
> >>>>
> >>>> # Change to directory from which qsub command was issued
> >>>>   cd $PBS_O_WORKDIR
> >>>>
> >>>> cat $PBS_NODEFILE
> >>>> #Call mpirun with 1 copy of the R interpreter. This will spawn 31
> >>> workers,
> >>>> inside the R script
> >>>> time mpirun -n 1 R --slave -f ParallelAnalysis.R
> >>>> ***End PBS script***
> >>>>
> >>>> ***Begin R Script***
> >>>> source("http://bioconductor.org/biocLite.R")
> >>>> #MPI stuff initialization
> >>>> library(Rmpi)
> >>>> library(foreach)
> >>>> library(doMPI)
> >>>> cl <- startMPIcluster(count=31) #call 31 clusterworkers/slaves
> >>>> registerDoMPI(cl)
> >>>> library(MEDIPS)
> >>>> library(BSgenome)
> >>>> .
> >>>> .
> >>>> *more R code; variable assignments etc; no mpi stuff here*
> >>>> .
> >>>> .
> >>>> #Following code will run 100 parallel iterations using the doMPI
> library
> >>>> loaded above and output results to the variable x. x is a table and
> >>> stores
> >>>> results from iterations as rows.
> >>>> x <-foreach(i=1:100,.combine='rbind') %dopar% {
> >>>> *stuff to do inside loop*
> >>>> }
> >>>> write.table(x, "output.tsv") #write x into file.
> >>>> ***End R script***
> >>>>
> >>>> The execution halts as soon as the libraries are loaded - I get the
> >>>> following error message repeatedly from both nodes (node 203 and node
> >>> 202)
> >>>>
> >>>> *[2:node203] unexpected disconnect completion event from dynamic
> process
> >>>> with rank=0 pg*
> >>>> *_id=kvs_17890_0 0x1fce600*
> >>>> *Assertion failed in file ../../dapl_conn_rc.c at line 1128: 0*
> >>>>
> >>>> I am not sure if this is an issue with the compilers or the script
> >>> itself.
> >>>> The script runs successfully without using mpi (using only 1 node).
> Any
> >>>> help would be highly appreciated.
> >>>>
> >>>> Thanks in advance,
> >>>> Srihari
> >>>>
> >>>> --
> >>>> Srihari Radhakrishnan
> >>>>
> >>>> Ph.D candidate
> >>>> Valenzuela Lab
> >>>> Iowa State University
> >>>>
> >>>>        [[alternative HTML version deleted]]
> >>>>
> >>>> _______________________________________________
> >>>> R-sig-hpc mailing list
> >>>> R-sig-hpc@r-project.org
> >>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >>>
> >>>
> >>>
> >>> ------------------------------
> >>>
> >>> Message: 3
> >>> Date: Mon, 21 Oct 2013 15:56:59 +0000
> >>> From: "Lockwood, Glenn" <glock@sdsc.edu>
> >>> To: "r-sig-hpc@r-project.org" <r-sig-hpc@r-project.org>
> >>> Subject: Re: [R-sig-hpc] Help with doMPI on multiple cores on a
> >>>        cluster
> >>> Message-ID: <5D6CCDC1-CBB6-4926-8022-B19EC68CDAC5@sdsc.edu>
> >>> Content-Type: text/plain; charset="us-ascii"
> >>>
> >>> I'd like to echo Steve's advice--try using OpenMPI instead.  I've had
> >>> innumerable problems trying to get mvapich2 (on which IntelMPI is
> based)
> >>> and Rmpi to work, and officially, Rmpi only supports openmpi and mpich.
> >>> It's an uphill battle.
> >>>
> >>> Also, you should be running mpirun with -n 1 (as you are already doing)
> >>> if you are calling R directly.  Doing anything else will cause multiple
> >>> master scripts to run, each spawning its own set of mpi ranks and
> leaving
> >>> you with a lot more MPI ranks than you want.  Some libraries provide
> >>> special wrappers that let you call them directly using mpirun -np 32
> (e.g.,
> >>> snow provies the RMPISNOW command), but these are unique to each
> library
> >>> whereas using mpirun -n1 is universal.
> >>>
> >>> Glenn
> >>>
> >>> On Oct 21, 2013, at 6:55 AM, Stephen Weston <
> stephen.b.weston@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Srihari,
> >>>>
> >>>> I suspect it's an MPI issue.  Are you able to run any other simple MPI
> >>>> programs successfully, and specifically, any using R with Rmpi?  From
> >>>> the error message, it appears that you're using Intel MPI, which I've
> >>>> never used.  I believe Rmpi is primarily tested with Open MPI, which
> >>>> is what I've always used with doMPI.  It would be interesting to see
> >>>> if you can run successfully using Open MPI, if that is possible for
> >>>> you.
> >>>>
> >>>> You'll probably need to look for help on an Intel MPI forum, although
> >>>> you may need to reduce the problem to something that doesn't use R.
> >>>>
> >>>> Here is a similar issue that I found on an Intel MPI forum:
> >>>>
> >>>>   http://software.intel.com/en-us/forums/topic/329053
> >>>>
> >>>> You could also try running without spawning, since that may be a
> >>>> problem for Intel MPI.  To do that, change the R script to use:
> >>>>
> >>>>   cl <- startMPIcluster()
> >>>>
> >>>> Also change the mpirun command in the PBS script to use '-n 32' or
> >>>> don't specify the -n option at all.  In that case, mpirun will start
> >>>> all of the workers as well as the master which may work better.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Steve Weston
> >>>>
> >>>> On Mon, Oct 21, 2013 at 9:42 AM, Srihari Radhakrishnan
> >>>> <srihari@iastate.edu> wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I've been trying to use the doMPI to run the iterations of a for loop
> >>> in
> >>>>> parallel (using the foreach package) on a cluster. However, I've been
> >>>>> running into issues - I think its the way I am running the R script,
> >>> but I
> >>>>> could be wrong. Here's the description of the problem.
> >>>>>
> >>>>> We use a PBS scheduler to submit jobs; my script uses 2 nodes (32
> >>> cores)
> >>>>> for now. I run 1 version of the R interpreter which internally calls
> 31
> >>>>> workers using R's mpi libraries. I produce below the PBS script, the
> R
> >>> code
> >>>>> (the relevant bits) and the error.
> >>>>>
> >>>>> ***Begin PBS Script***
> >>>>> #!/bin/bash
> >>>>>
> >>>>> #PBS  -o BATCH_OUTPUT
> >>>>> #PBS  -e BATCH_ERRORS
> >>>>>
> >>>>> #PBS -lnodes=2:ppn=16:compute,walltime=12:00:00
> >>>>>
> >>>>> # Change to directory from which qsub command was issued
> >>>>>  cd $PBS_O_WORKDIR
> >>>>>
> >>>>> cat $PBS_NODEFILE
> >>>>> #Call mpirun with 1 copy of the R interpreter. This will spawn 31
> >>> workers,
> >>>>> inside the R script
> >>>>> time mpirun -n 1 R --slave -f ParallelAnalysis.R
> >>>>> ***End PBS script***
> >>>>>
> >>>>> ***Begin R Script***
> >>>>> source("http://bioconductor.org/biocLite.R")
> >>>>> #MPI stuff initialization
> >>>>> library(Rmpi)
> >>>>> library(foreach)
> >>>>> library(doMPI)
> >>>>> cl <- startMPIcluster(count=31) #call 31 clusterworkers/slaves
> >>>>> registerDoMPI(cl)
> >>>>> library(MEDIPS)
> >>>>> library(BSgenome)
> >>>>> .
> >>>>> .
> >>>>> *more R code; variable assignments etc; no mpi stuff here*
> >>>>> .
> >>>>> .
> >>>>> #Following code will run 100 parallel iterations using the doMPI
> >>> library
> >>>>> loaded above and output results to the variable x. x is a table and
> >>> stores
> >>>>> results from iterations as rows.
> >>>>> x <-foreach(i=1:100,.combine='rbind') %dopar% {
> >>>>> *stuff to do inside loop*
> >>>>> }
> >>>>> write.table(x, "output.tsv") #write x into file.
> >>>>> ***End R script***
> >>>>>
> >>>>> The execution halts as soon as the libraries are loaded - I get the
> >>>>> following error message repeatedly from both nodes (node 203 and node
> >>> 202)
> >>>>>
> >>>>> *[2:node203] unexpected disconnect completion event from dynamic
> >>> process
> >>>>> with rank=0 pg*
> >>>>> *_id=kvs_17890_0 0x1fce600*
> >>>>> *Assertion failed in file ../../dapl_conn_rc.c at line 1128: 0*
> >>>>>
> >>>>> I am not sure if this is an issue with the compilers or the script
> >>> itself.
> >>>>> The script runs successfully without using mpi (using only 1 node).
> Any
> >>>>> help would be highly appreciated.
> >>>>>
> >>>>> Thanks in advance,
> >>>>> Srihari
> >>>>>
> >>>>> --
> >>>>> Srihari Radhakrishnan
> >>>>>
> >>>>> Ph.D candidate
> >>>>> Valenzuela Lab
> >>>>> Iowa State University
> >>>>>
> >>>>>       [[alternative HTML version deleted]]
> >>>>>
> >>>>> _______________________________________________
> >>>>> R-sig-hpc mailing list
> >>>>> R-sig-hpc@r-project.org
> >>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >>>>
> >>>> _______________________________________________
> >>>> R-sig-hpc mailing list
> >>>> R-sig-hpc@r-project.org
> >>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >>>
> >>>
> >>>
> >>> ------------------------------
> >>>
> >>> _______________________________________________
> >>> R-sig-hpc mailing list
> >>> R-sig-hpc@r-project.org
> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >>>
> >>>
> >>> End of R-sig-hpc Digest, Vol 61, Issue 4
> >>> ****************************************
> >>>
> >>
> >>
> >>
> >> --
> >> Srihari Radhakrishnan
> >>
> >> Ph.D candidate
> >> Valenzuela Lab
> >> Iowa State University
> >>
> >> Website: http://www.public.iastate.edu/~srihari/
> >>
> >
> >
> >
> > --
> > Srihari Radhakrishnan
> >
> > Ph.D candidate
> > Valenzuela Lab
> > Iowa State University
> >
> > Website: http://www.public.iastate.edu/~srihari/
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-hpc mailing list
> > R-sig-hpc@r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>
>


-- 
Srihari Radhakrishnan

Ph.D candidate
Valenzuela Lab
Iowa State University

Website: http://www.public.iastate.edu/~srihari/

	[[alternative HTML version deleted]]

