Hello Ryan,

You may be interested in the function sclapply(...) located in the
HTSeqGenie package. sclapply is a multicore dispatcher that accepts 3 main
arguments (inext, fun, max.parallel.jobs). The data produced by the
function inext, executed in the main thread, is dispatched to fun(),
executed in a children thread. A built-in scheduler controls the maximal
number of threads.

In HTSeqGenie, inext(...) is typically an iterator to read chunks of FastQ
reads, which are passed to a function processing the FastQ reads (for
counting, QC, alignment...) in a children thread. sclapply(...) enables
multicore processing of iterator flows and offers performance gains almost
proportional to the number of cores. Moreover, the function is robust and
contains extra arguments to handle exceptions and periodical tracing (e.g.
to check memory usage).

Hope this can help,

Cheers,

Greg


On Mon, Sep 30, 2013 at 5:00 PM, Ryan <rct@thompsonclan.org> wrote:

> Hi all,
>
> I have previously written an Rscript to read, filter, and write large
> fastq files using FastqSteamer to read. Through some complicated tricks, I
> was able to get the input to happen in parallel with the processing and
> output (using parallel::mcparallel and friends). In other words, while my
> script was processing and writing out the nth block of reads, another
> process was reading the (n+1)th block of reads at the same time. This
> almost doubled the speed of my script (the server had sufficient I/O
> bandwidth to parallelize reads and writes to disk). Since then, I've been
> wanting to generalize this pattern, and I have just now made a working
> proof of concept. It is a wrapper for FastqStreamer that runs in a separate
> process and uses parallel:::sendMaster to send each block to the main
> script, and then calls yield on the FastqStreamer to preload the next block
> while the script is processing the current one. You can view and download
> the script here:
>
> https://gist.github.com/**DarwinAwardWinner/6771922<https://gist.github.com/DarwinAwardWinner/6771922>
>
> I have strategically placed print statementsin the code in order to
> demonstrate that preloading is happening. For example, I get the following
> when I run the script on my machine:
>
> CHILD: Preloaded 1 yields.
> CHILD: Sent 1 yields.
> CHILD: Preloaded 2 yields.
> CHILD: Sent 2 yields.
> MAIN: Received 1 yields.
> MAIN: Processing reads
> CHILD: Preloaded 3 yields.
> MAIN: Processed 1 yields.
> CHILD: Sent 3 yields.
> MAIN: Received 2 yields.
> MAIN: Processing reads
> CHILD: Preloaded 4 yields.
> MAIN: Processed 2 yields.
> CHILD: Sent 4 yields.
> MAIN: Received 3 yields.
> MAIN: Processing reads
> CHILD: Preloaded 5 yields.
> MAIN: Processed 3 yields.
> CHILD: Sent 5 yields.
> MAIN: Received 4 yields.
> MAIN: Processing reads
> CHILD: Preloaded 6 yields.
> MAIN: Processed 4 yields.
> CHILD: Sent 6 yields.
> MAIN: Received 5 yields.
> MAIN: Processing reads
> MAIN: Processed 5 yields.
> MAIN: Received 6 yields.
> MAIN: Processing reads
> MAIN: Processed 6 yields.
>
> In the script, the child is reading the fastq file, and the main process
> is doing the "calculation" (which is just a sleep). As you can see, the
> child is always a step or two ahead of the main script, so that whenever
> the main script asks for the next yield, it gets it immediately instead of
> waiting for the child to read from the disk.
>
> So, is this kind of feature appropriate for inclusion into BioConductor?
>
> -Ryan Thompson
>
> ______________________________**_________________
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/**listinfo/bioc-devel<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
>

	[[alternative HTML version deleted]]

