[R-sig-ME] How to obtain Type 2 like p-values for effects?

[Henrik Singmann]

Dear John,

The difference between car::Anova() and my mixed() function is (as we 
already discussed off list) that mixed() uses  a likelihood ratio (LR) 
test approach (i.e., comparing two models) and Anova() uses the Wald 
tests. As the approach is somewhat different my question is if there is 
a way to obtain reasonable "type-II" tests using the model comparison 
approach. Perhaps it would have been better to frame the question in 
such a way.
Actually I checked both of my solutions against the development versions 
of Anova() and they don't match. However, solution (2) comes close. In 
contrast, the "type-III" tests perfectly match the results obtained with 
car::Anova().

Do you think the LR approach is generally flawed when compared with a 
Wald test approach? Or is there a way to obtain reasonable test for 
effetcs obeying marginality using LR?

Best,
Henrik

John Fox schrieb:
> Dear Henrik,
>
> The Anova() function in the car package already does this for Wald tests of the fixed effects in a mixed model, using pbkrtest to compute df (in the development version of the car package on R-Forge). We define the "type-II" test as the maximally powerful Wald test for the hypothesis in question obeying marginality.
>
> I hope this helps,
>   John
>
> ------------------------------------------------
> John Fox
> Sen. William McMaster Prof. of Social Statistics
> Department of Sociology
> McMaster University
> Hamilton, Ontario, Canada
> http://socserv.mcmaster.ca/jfox/
> 	
> On Wed, 22 Aug 2012 20:22:40 +0200
>   Henrik Singmann <henrik.singmann at psychologie.uni-freiburg.de> wrote:
>> Dear all,
>>
>> I am currently trying to add the possibility to obtain Type 2 like p-values for all effects in a mixed model to function mixed() in package afex (using pbkrtest::KRmodcomp see: https://stat.ethz.ch/pipermail/r-sig-mixed-models/2012q3/018946.html) but am unsure on how to implement this correctly.
>>
>> I came up with two possible solutions which both seem to have problems.
>>
>> (1) Fit one full model and contrast a model for each effect with this full model. Each submodel is created by subtracting the effect of interest and all higher order effects from the model (e.g., when interested in a main effect, the submodel contains all main effects but the one of interest and no interactions).
>>
>> (2) Fit a full model for each order of effects (i.e., one full model for all main effects, one full model for all two-way interactions, ...) and contrast a model for each effect with the full model of the corresponding order (e.g., when interested in a main effect, the submodel contains all main effects but the one of interest and no interactions and the full model contains all main effects and no interactions).
>>
>> Both solutions seem to have problems.
>> The first solution seems rather dubious, why would you want to compare two models to test for a single effect when in fact more than just this effect is missing, namely all the higher order effects. Consequently, this solution does produces small p-values and many significant effects. For example, comparing a full model with many interactions with a model for the main effects in which only those are present almost inevitably needs to be significant.
>> The second solution seems problematic from another perspective, as it simply ignores the higher order effects. This contrasts with classical ANOVA in which even the lower order effects are tested against the MSE of the full model. Consequently, using the second approach the lower order effects of a model with interactions are identical to a model fitted without those interactions.
>>
>> Any ideas would be really appreciated.
>>
>> (Sidenote: I obtain type 3 tests by contrasting the full model with a model in which only the relevant effect is missing, ignoring the order of the effects)
>>
>> Cheers,
>> Henrik
>>
>> -- 
>> Dipl. Psych. Henrik Singmann
>> PhD Student
>> Albert-Ludwigs-Universität Freiburg, Germany
>> http://www.psychologie.uni-freiburg.de/Members/singmann
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

-- 
Dipl. Psych. Henrik Singmann
PhD Student
Albert-Ludwigs-Universität Freiburg, Germany
http://www.psychologie.uni-freiburg.de/Members/singmann