  Dear all,

  I am trying to pre-process (bg correction, quantile normalization,
summarization) the readings in a large number (~1,000s) of large microarray
chips (~10^6 probes).
  As far as I can tell, pre-processing functions in most packages will load
data from all chips at once which, in this case, is infeasible.
  In addition, I'd like to have flexibility in how to do summarization to
the gene or exon level at the last pre-processing step.

  If I understand correctly, it should be possible to do all the processing
without loading the entire data set at once as described below.
  Can anyone comment whether that sounds sensible?

  The main questions are:
  1) In the RMA background correction step, data from each chip is used
independently from data in other chips, correct?
     1a) If so, the background corrected intensities for each chip can be
computed using the affyio::read.celfile and
preprocessCore::rma.background.correct functions, correct?

  2) In the quantile normalization step: which probes are included in the
ordered vector of intensities used to construct the "reference distribution
of intensities" shared across all probes?
     Specifically, MM probes are NOT included, but what about control
probes?
     For HTA2,0 the probeset types are control->affx, control->affx->asc,
control->affx->bac_spike, ..., normgene->exon, normgene->intron. Which of
these are included in quantile normalization?

  3) When a CEL file is read using affyio::read.celfile, in what order are
the mean intensities included in the INTENSITY$MEAN vector:
     3a) X first as in (X=0, Y=0), (X=1, Y=0), ..., (X=max.X, Y=0), (X=0,
Y=1), (X=1, Y=1), ..., (X=max.X, Y=1), ..., (X=0, Y=max.Y), (X=1, Y=max.Y),
..., (X=max.X, Y=max.Y)?
     3b) Y first as in (X=0, Y=0), (X=0, Y=1), ..., (X=0, Y=max.Y), (X=1,
Y=0), (X=1, Y=1), ..., (X=1, Y=max.Y), ..., (X=max.X, Y=0), (X=max.X, Y=1),
..., (X=max.X, Y=max.Y)?
     3c) Some different order?

  4) In the RMA summarization step, the normalized intensities in a chip
are processed independently from data in other chips, correct?
     The subColSummarize in preprocessCore can be used to do this (as long
as I can group probes into probesets or genes), correct?


  Any help appreciated,

  Thanks,

  G. Rocha



-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  "Distributed" RMA normalization algorithm:


  a) Background correction:
     For each chip, CEL file can be read using affyio::read.celfile and the
background correction can be done using
preprocessCore::rma.background.correct and save intensities in a separate
(binary) file
     Is this any different than what is done internally at rma???

  b) Quantile normalization:
     This is the more involved step as it requires data from all chips.
     But it is possible to avoid loading the entire data by doing two
passes through the data:
     Pass 1) Open file with bg corrected for each chip and sum ORDERED
intensities along the way; once finished summing, divide by n_chips to get
ordered intensities in a ordered vector of "reference intensities";
     Pass 2) For each chip, Open file with bg corrected measurements,
compute rank for each probe and substitute it with the corresponding rank
on the vector of "reference intensities".
             Save bg-corrected, normalized probe level intensities for each
chip separately.

  c) Summarization:
     For each chip, open file of bg-corrected, normalized probe level
intensities created in (b).
     Summarize to probeset, gene, exon, junction level using your favorite
version of preprocessCore::subColSummarize.


-- 
Guilherme V. Rocha
gvrocha@gmail.com

	[[alternative HTML version deleted]]

