[R] observed power

[Peter Dalgaard BSA]

David W <dwinsemius at home.com> writes:

> Two or three years ago there was an extended discussion on
> sci.stat.consult concerning "post-hoc power analysis". If
> memory serves (since DejaNews is not currently allowing
> searches back that far), it was power estimation using the
> sample estimate of the variance done after conducting a
> "failed" experiment . Some of the contributors did not seem
> to understand that the variance was subject to sampling
> error. Using the observed interim sample variance to
> determine a plausible number of additional subjects would
> seem of some use after an experiment of insufficient size
> had testing consistent with the null. (With appropriate
> regard for adjusting alpha after multiple tests.)  It
> doesn't need to be distinguished by a separate term, and any
> reasonable interpretation would already be subsumed under
> existing interim analysis strategies.
...
> > | What's "observed power"? If you mean the item that SPSS has by that
> > | name, I think you first have to convince us that that is a sensible
> > | thing to calculate...
> > 
> > If you ever do find out, Peter, let me know too, please.  I was puzzled, but
> > a bit worried about showing my ignorance...   I can only imagine it means an
> > estimate of the non-centrality parameter in which case it is a sensible
> > thing to have available since it is essentially the signal-to-noise ratio.
> > Actually the MLE is the F-statistic but the maximum *marginal* likelihood
> > estimate (based on the marginal distribution of the F-statistic itself) is
> > of more interest as it is closer to unbiased.  In the case of the multiple
> > correlation coefficient, for example, this is (practically) what people call
> > the "adjusted R^2" statistic, where the adjustment is essentially a bias
> > correction.  You can come up with simple analogues for non-central
> > chi-squared and non-central F of course, but they are again just simple
> > linear adjustments, unless you really want to get flash.  (I wrote a couple
> > of papers on this stuff in the 70s so I have a kind of nostalgic affinity
> > for it...)
> > 
> > I would be more interested in these quantities optionally appearing
> > routinely on summary tables than, for example, the cute 'significance
> > stars'.  But as for calling them the "observed power", I would definitely
> > caution against that.  It encourages entirely the wrong idea of what power
> > really is.  (For example, it is a function, not a quantity, and you don't
> > ever "observe" it in practice.)

My understanding of what SPSS does is that it uses an estimate of the
noncentrality and of the variance and plugs it into the usual power
formulas. So for a t test with p just under 0.05, it comes out with an
"observed power" of about 0.50 (so it's not really significant or
what?). Worse, for near-zero effects it comes out with a very low
power, leading to pernicious nonsense of the type "we (or they) didn't
see an effect, but the study had low power to detect it". One common
task in consulting is to explain to people that power calculations
belong *before* the experiment was conducted and that once data are
collected, confidence intervals make better sense.

I'm not quite sure exactly what SPSS is doing to get the noncentrality
parameters, but finding an example with p=0.05 and obs.power=0.50
exactly, would indicate that it is using the F statistic itself as the
estimate. 

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)             FAX: (+45) 35327907
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._