[Bioc-devel] Proof-of-concept parallel preloading FastqStreamer

[Ryan]

Hi all,

I have previously written an Rscript to read, filter, and write large 
fastq files using FastqSteamer to read. Through some complicated tricks, 
I was able to get the input to happen in parallel with the processing 
and output (using parallel::mcparallel and friends). In other words, 
while my script was processing and writing out the nth block of reads, 
another process was reading the (n+1)th block of reads at the same time. 
This almost doubled the speed of my script (the server had sufficient 
I/O bandwidth to parallelize reads and writes to disk). Since then, I've 
been wanting to generalize this pattern, and I have just now made a 
working proof of concept. It is a wrapper for FastqStreamer that runs in 
a separate process and uses parallel:::sendMaster to send each block to 
the main script, and then calls yield on the FastqStreamer to preload 
the next block while the script is processing the current one. You can 
view and download the script here:

https://gist.github.com/DarwinAwardWinner/6771922

I have strategically placed print statementsin the code in order to 
demonstrate that preloading is happening. For example, I get the 
following when I run the script on my machine:

CHILD: Preloaded 1 yields.
CHILD: Sent 1 yields.
CHILD: Preloaded 2 yields.
CHILD: Sent 2 yields.
MAIN: Received 1 yields.
MAIN: Processing reads
CHILD: Preloaded 3 yields.
MAIN: Processed 1 yields.
CHILD: Sent 3 yields.
MAIN: Received 2 yields.
MAIN: Processing reads
CHILD: Preloaded 4 yields.
MAIN: Processed 2 yields.
CHILD: Sent 4 yields.
MAIN: Received 3 yields.
MAIN: Processing reads
CHILD: Preloaded 5 yields.
MAIN: Processed 3 yields.
CHILD: Sent 5 yields.
MAIN: Received 4 yields.
MAIN: Processing reads
CHILD: Preloaded 6 yields.
MAIN: Processed 4 yields.
CHILD: Sent 6 yields.
MAIN: Received 5 yields.
MAIN: Processing reads
MAIN: Processed 5 yields.
MAIN: Received 6 yields.
MAIN: Processing reads
MAIN: Processed 6 yields.

In the script, the child is reading the fastq file, and the main process 
is doing the "calculation" (which is just a sleep). As you can see, the 
child is always a step or two ahead of the main script, so that whenever 
the main script asks for the next yield, it gets it immediately instead 
of waiting for the child to read from the disk.

So, is this kind of feature appropriate for inclusion into BioConductor?

-Ryan Thompson