Let's try to get an overview of these issues together for discussion at
developer day.
I will start a wiki node for developer day and drop this exchange within.

On Fri, Jul 6, 2012 at 10:33 PM, Michael Lawrence <lawrence.michael@gene.com
> wrote:

> Hi Kasper,
>
> I think Herve was going to look into some optimizations to the lapply
> behavior over the next release cycle. But I like the idea of leveraging
> parallelism better. The user would specify options(mc.cores=), so that we
> could avoid having to pass down more arguments.
>
> In the bigger picture, it would be nice if there were some general way for
> the user to request parallelism. Setting options(mc.cores=) is a nice
> approach, but it is specific to mclapply. I'm aware that mclapply has the
> advantage of the slaves inheriting the session from the master, but often
> more than one computer is available for a task. I admit that in this case
> of overlap computation, the network overhead may not be worth it.  But
> methods that e.g. operate over a BamFileList might benefit a lot.
>
> I wonder if the general clusterApply mechanism could be leveraged. The user
> would specify a global default cluster using setDefaultCluster().
> Currently, the cluster implementations beyond the mclapply forking is
> pretty limited. One possible direction would be to implement a parallel
> package backend using the new CRAN BatchJobs package, which supports
> submitting jobs to LSF, SSH, etc, clusters. That API is currently
> asynchronous, so some work would need to be done to make it behave
> synchronously.
>
> Michael
>
> On Fri, Jul 6, 2012 at 8:51 AM, Kasper Daniel Hansen <
> kasperdanielhansen@gmail.com> wrote:
>
> > This is about the findOverlaps method for (query = "GenomicRanges",
> > subject = "GenomicRanges") in GenomicRanges.
> >
> > This function is really slow when the set of distinct seqnames is
> > really big.  Example
> >
> > library(BSgenome.Amellifera.BeeBase.assembly4)
> > Un <- Amellifera$GroupUn
> > gr <- GRanges(seqnames = names(Un),
> >               ranges= IRanges(start = 1 , width = width(Un)))
> > length(gr)
> >
> > ## Only 9244 in length
> >
> > system.time(findOverlaps(gr[1], gr))
> >   user  system elapsed
> >  297.202   0.021 297.279
> >
> > Pretty slow for finding overlaps between a Granges with length 1 and a
> > Granges with length roughly 10000.
> >
> > This is because the function essentially does an lapply over distinct
> > seqnames.  I raised this issue a while ago, and Michael said that he
> > might consider building an IntervalTree over both seqnames and ranges.
> >  So this is really only an issue in organisms with many small contigs.
> >  However, in the mean time, I would appreciate making the function
> > mclapply-aware, which is pretty simple and at least makes the function
> > roughly #cores faster.
> >
> > My fix (which I have been using for a while) adds
> >   mc.cores = 1, mc.preschedule = TRUE
> > to the specific method as well as
> >
> >         matchMatrix <- do.call(rbind, mclapply(commonSeqnames,
> >                                                function(seqnm) {
> > <SNIP>
> >                                                }, mc.cores = mc.cores,
> > mc.preschedule = mc.preschedule))
> >
> > inside the methods definition.  Also needed is making the package
> > depend on parallel.
> >
> > I am raising this issue because I am submitting a package containing
> > this fix, and I felt it might be good to propagate.  I can commit it
> > to subversion as well, if there is any interest.
> >
> > Best,
> > Kasper
> >
> > _______________________________________________
> > Bioc-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>

	[[alternative HTML version deleted]]

