[R] Power analysis for MANOVA?

[Adam D. I. Kramer]

Thanks for the response, Stephan.

Really, I am trying to say, "My result is insignificant, my effect sizes are
tiny, you may want to consider the possibility that there really are no
meaningful differences." Computing post-hoc power makes a bit stronger of a
claim in this setting.

My real goal in this case was to put a single line on a poster that says,
"Significance using our estimates would require N observations, which is
larger than the population." I am trying to solve for N.  N in this case is
a sort of effect size.  In this case, it is indeed a simple transformation
of Pillai's V and the p-value for the study, and I do not intend to suggest
that it is anything more than that.  However, I believe that the latter
effect size makes a much more compelling case, given that a lot of people
(such as yourself) don't have much experience with Pillai's V.

--Adam

On Wed, 28 Jan 2009, Stephan Kolassa wrote:

> Hi Adam,
>
> first: I really don't know much about MANOVA, so I sadly can't help you 
> without learning about it an Pillai's V... which I would be glad to do, but I 
> really don't have the time right now. Sorry!
>
> Second: you seem to be doing a kind of "post-hoc power analysis", "my result 
> isn't significant, perhaps that's due to low power? Let's look at the power 
> of my experiment!" My impression is that "post-hoc power analysis" and its 
> interpretation is, shall we say, not entirely accepted within the statistical 
> community, see:
>
> Hoenig, J. M., & Heisey, D. M. (2001, February). The abuse of power: The 
> pervasive fallacy of power calculations for data analysis. The American 
> Statistician, 55 (1), 1-6
>
> And this:
> http://staff.pubhealth.ku.dk/~bxc/SDC-courses/power.pdf
>
> However, I am sure that lots of people can discuss this more competently than 
> me...
>
> Best wishes
> Stephan
>
>
> Adam D. I. Kramer schrieb:
>> 
>> On Mon, 26 Jan 2009, Stephan Kolassa wrote:
>> 
>>> My (and, judging from previous traffic on R-help about power analyses,
>>> also some other people's) preferred approach is to simply simulate an
>>> effect size you would like to detect a couple of thousand times, run your
>>> proposed analysis and look how often you get significance.  In your simple
>>> case, this should be quite easy.
>> 
>> I actually don't have much experience running monte-carlo designs like
>> this...so while I'd certainly prefer a bootstrapping method like this one,
>> simulating the effect size given my constraints isn't something I've done
>> before.
>> 
>> The MANOVA procedure takes 5 dependent variables, and determines what
>> combination of the variables best discriminates the two levels of my
>> independent variable...then the discrimination rate is represented in the
>> statistic (Pillai's V=.00019), which is then tested (F[5,18653] = 0.71). 
>> So
>> coming up with a set of constraints that would produce V=.00019 given my
>> data set doesn't quite sound trivial...so I'll go for the "par" library
>> reference mentioned earlier before I try this.  That said, if anyone can
>> refer me to a tool that will help me out (or an instruction manual for 
>> RNG),
>> I'd also be much obliged.
>> 
>> Many thanks,
>> Adam
>> 
>> 
>>> 
>>> HTH,
>>> Stephan
>>> 
>>> 
>>> Adam D. I. Kramer schrieb:
>>>> Hello,
>>>>
>>>>     I have searched and failed for a program or script or method to
>>>> conduct a power analysis for a MANOVA. My interest is a fairly simple 
>>>> case
>>>> of 5 dependent variables and a single two-level categorical predictor
>>>> (though the categories aren't balanced).
>>>>
>>>>     If anybody happens to know of a script that will do this in R, I'd
>>>> love to know of it! Otherwise, I'll see about writing one myself.
>>>>
>>>>     What I currently see is this, from help.search("power"):
>>>> 
>>>> stats::power.anova.test
>>>>                         Power calculations for balanced one-way
>>>>                         analysis of variance tests
>>>> stats::power.prop.test
>>>>                         Power calculations two sample test for
>>>>                         proportions
>>>> stats::power.t.test     Power calculations for one and two sample t
>>>>                         tests
>>>>
>>>>     Any references on power in MANOVA would also be helpful, though of
>>>> course I will do my own lit search for them myself.
>>>> 
>>>> Cordially,
>>>> Adam D. I. Kramer
>>>> 
>>>> ______________________________________________
>>>> R-help at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide 
>>>> http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible code.
>>>> 
>>> 
>>> 
>> 
>
>