Hi Chris,

Thanks for sharing your thoughts.

The reviewer used the heterogeneity that I observed in my study for the power analysis. I understand what you have descried. And I agree that with the sample size I have, I do not have enough power to detect the heterogeneity that I observed with significance.

But if let's say I have enough sample size as calculated by the power analysis, then I will have 80% power to detect the heterogeneity, would it be true that I will almost very unlikely to declare homogeneity among study sites, so that I would almost never be able to combine study sites? This goes to the general thinking that if you have a sample size large enough, you will always be able to make any difference statistically significant...

On the the hand, making a statistical inference using any statistical test (including Mantel Haenszel test), I though, is always valid regardless of sample size. For the heterogeneity test, I am just doing that -- making a statistical inference with the p value from Mantel Haenszel test.  I am not sure if it is correct that it is mandatory to perform a power analysis before attempting a statistical test.


Please share your thoughts...

Thanks

John


________________________________
 From: Christopher W. Ryan <cryan@binghamton.edu>

Sent: Tuesday, November 12, 2013 6:53 PM
Subject: Re: [R] power analysis is applicable or not


John--

Well, my simple-minded way of thinking about these issues goes something
like this:

You want to know if there is heterogeneity. You gather some data and do
your MH analysis. You never know *for sure* whether there is *really*
heterogeneity in your population; all you know is whether there is any
in your sample--you concluded there was not. Your reviewer calculated
that with the sample size you used, *even if there was heterogeneity in
your population* (unknowable by you or anyone else) then your sample
size only had a 50% probability of detecting it (a 50% probability of
coming up with a p < 0.05).  Meaning there *could have been*
heterogeneity there, at a 0.05 signficance level, and you *would* have
seen it, *if* your sample size was larger.

It's when you come up with a "non-significant result" that the issue of
power is most relevant. If you already have a "significant" result, then
yes, your sample size was large enough to show a significant result.

An important question is: what *magnitude* of heterogeneity did your
reviewer assume he/she was looking for when he/she did the power
calculation?  And is that magnitude meaningful?

All this being said, power calculations are best done before recruiting
subjects or gathering data.

--Chris Ryan
SUNY Upstate Medical University
Binghamton, NY

array chip wrote:
> Hi, this is a statistical question rather than a pure R question. I have got many help from R mailing list in the past, so would like to try here and appreciate any input:
> 
> I conducted Mantel-Haenszel test to show that the performance of a diagnostic test did not show heterogeneity among 4 study sites, i.e. Mantel Haenszel test p value > 0.05,  so that I could conduct a meta-analysis by combining data of all 4 study sites. 
> 
> Now one of the reviewers for the manuscript did a powering analysis for Mantel Haneszel test showing that with the sample sizes I have, the power for Mantel Haeszel test was only 50%. So he argued that I did not have enough power for Mantel Haenszel test.
> 
> My usage of Mantel Haenszel was NOT to show a significant p value, instead a non-sginificant p value was what I was looking for because non-significant p value indicate NO heterogeneity among study sites. Powering analysis in general is to show whether you have enough sample size to ensure a statistical significant difference can be seen with certain likelihood. But this is not how I used Mantel Haenszel test. So I think in my scenario, the power analysis is NOT applicable because I am simply using the test to demonstrate a non-significant p value.
> 
> Am I correct on this view?
> 
> Thanks and appreciate any thoughts.
> 
> John
>     [[alternative HTML version deleted]]
> 
> 
> 
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

> 
	[[alternative HTML version deleted]]

