[BioC] FW: DESeq query regarding rerVarA and resVarB columns

[Simon Anders]

Hi Natasha

> In DeSeq, once one uses the nbinom test (Rv2.13, DeSeq v1.4.1) for
> differential expression, the vignette states that to check for false
> positives one needs to look at the resVarA and resVarB columns.
>
> If one (or perhaps both) have large values then they are potentially
> outliers.
>
> My question then is how large is large? How much above the value of 1 is
> considered to be large?
>
> In my case, I have 2 replicates in each group.

Recommending a good cut-off threshold turned out to be rather 
challenging. For two replicates, for example, 5 might be a "safe" value 
that will, however, also remove many valid hits, while something around 
10 or 20 might already let pass too much.

This is why we overhauled this aspect in the current devel version of DESeq.

There, we now use the maximum of the per-gene estimate for the 
dispersion (or variance) and the fitted value. Hence, if the residual is 
above 1, the test becomes more conservative, and that gradually, so that 
the problems with a hard cut-off are circumvented.

This costs a little bit of power, but overall seems to work very well.

Have a look at the updated vignette in the devel version for details.

   Simon