[R] power analysis is applicable or not

[David Winsemius]

On Nov 12, 2013, at 7:42 PM, array chip wrote:

> Hi Chris,
> 
> Thanks for sharing your thoughts.
> 
> The reviewer used the heterogeneity that I observed in my study for the power analysis. I understand what you have descried. And I agree that with the sample size I have, I do not have enough power to detect the heterogeneity that I observed with significance.

You said that: "Now one of the reviewers for the manuscript did a powering analysis for Mantel Haneszel test showing that with the sample sizes I have, the power for Mantel Haeszel test was only 50%. So he argued that I did not have enough power for Mantel Haenszel test."

This is rather interesting in its own right. Generally if you find that the p-value is 0.05 then you are exactly at the point where the post-hoc power (the power calculated on the basis of the observed differences and variances)  will be 0.50. In other words if you are right at the tipping point then a small perturbation in the data will tip you either way. And yet you said the p value was > 0.05 (although you didn't say how much greater.)  So I would say your power was certainly less and possibly materially less than 0.50. (This is dodging the question of power to detect exactly "what?". So far we have neither a discssion of the underlying scientific question nor any specifics.)

> 
> But if let's say I have enough sample size as calculated by the power analysis, then I will have 80% power to detect the heterogeneity, would it be true that I will almost very unlikely to declare homogeneity among study sites, so that I would almost never be able to combine study sites?

(That was incoherent.)

> This goes to the general thinking that if you have a sample size large enough, you will always be able to make any difference statistically significant...
> 
> On the the hand, making a statistical inference using any statistical test (including Mantel Haenszel test), I though, is always valid regardless of sample size.

That is just wrong.

> For the heterogeneity test, I am just doing that -- making a statistical inference with the p value from Mantel Haenszel test.  I am not sure if it is correct that it is mandatory to perform a power analysis before attempting a statistical test.

The question is whether you are justified in ignoring (or leaving out of the analysis) covariates that you thought a priori had a good chance of confounding the relationship of the predictors of interest on the outcome of interest. It appears that you have insufficient such justication. (And some statsiticians of excelletn repute would say you never have justification to do so regardless of any testing.)

-- 
David.
> 
> Please share your thoughts...
> 
> Thanks
> 
> John
> From: Christopher W. Ryan <cryan at binghamton.edu>
> To: array chip <arrayprofile at yahoo.com> 
> Sent: Tuesday, November 12, 2013 6:53 PM
> Subject: Re: [R] power analysis is applicable or not
> 
> John--
> 
> Well, my simple-minded way of thinking about these issues goes something
> like this:
> 
> You want to know if there is heterogeneity. You gather some data and do
> your MH analysis. You never know *for sure* whether there is *really*
> heterogeneity in your population; all you know is whether there is any
> in your sample--you concluded there was not. Your reviewer calculated
> that with the sample size you used, *even if there was heterogeneity in
> your population* (unknowable by you or anyone else) then your sample
> size only had a 50% probability of detecting it (a 50% probability of
> coming up with a p < 0.05).  Meaning there *could have been*
> heterogeneity there, at a 0.05 signficance level, and you *would* have
> seen it, *if* your sample size was larger.
> 
> It's when you come up with a "non-significant result" that the issue of
> power is most relevant. If you already have a "significant" result, then
> yes, your sample size was large enough to show a significant result.
> 
> An important question is: what *magnitude* of heterogeneity did your
> reviewer assume he/she was looking for when he/she did the power
> calculation?  And is that magnitude meaningful?
> 
> All this being said, power calculations are best done before recruiting
> subjects or gathering data.
> 
> --Chris Ryan
> SUNY Upstate Medical University
> Binghamton, NY
> 
> array chip wrote:
> > Hi, this is a statistical question rather than a pure R question. I have got many help from R mailing list in the past, so would like to try here and appreciate any input:
> > 
> > I conducted Mantel-Haenszel test to show that the performance of a diagnostic test did not show heterogeneity among 4 study sites, i.e. Mantel Haenszel test p value > 0.05,  so that I could conduct a meta-analysis by combining data of all 4 study sites. 
> > 
> > Now one of the reviewers for the manuscript did a powering analysis for Mantel Haneszel test showing that with the sample sizes I have, the power for Mantel Haeszel test was only 50%. So he argued that I did not have enough power for Mantel Haenszel test.
> > 
> > My usage of Mantel Haenszel was NOT to show a significant p value, instead a non-sginificant p value was what I was looking for because non-significant p value indicate NO heterogeneity among study sites. Powering analysis in general is to show whether you have enough sample size to ensure a statistical significant difference can be seen with certain likelihood. But this is not how I used Mantel Haenszel test. So I think in my scenario, the power analysis is NOT applicable because I am simply using the test to demonstrate a non-significant p value.
> > 
> > Am I correct on this view?
> > 
> > Thanks and appreciate any thoughts.
> > 
> > John
> >     [[alternative HTML version deleted]]
> > 
> > 
> > 
> > ______________________________________________
> > R-help at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> 
> > 
> 
> 

David Winsemius
Alameda, CA, USA