[R-meta] how to interpreat a significant moderator variable but in-significant F test

[Viechtbauer, Wolfgang (SP)]

Dear Martin,

This can happen and it can go either way: 1) non-significant omnibus test combined with one or more significant contrasts with the reference level and 2) significant omnibus test combined with no significant contrasts with the reference level.

1) can happen when many pairs of levels do not differ from each other except for a few. Then the power of the omnibus test is 'dragged down' by only a few levels differing from each other. Note also that the one significant contrast just barely makes it below .05.

2) can happen when the pairs of levels that actually differ from each other are not contrasts with the reference level. Say there are three levels, A, B, and C, and let A be the reference level. Say A = 0, B = -0.5 and C = 0.5. Then maybe the A-B and A-C contrasts are not significant, but B-C could very well be, but you don't see this in the output directly since A is the reference level (one could of course easily test the B-C contrast based on the model). The omnibus test is not affected by the level chosen as the reference level and might pick up that large B-C difference.

So, in the end, you should decide on your testing strategy beforehand. You could: 1) only look at individual contrasts when the omnibus test is significant or 2) ignore the omnibus test and look at the individual contrasts. If you go with 2), then you might want to consider some adjustment for multiple testing (which one could also use with the first strategy, but this might be less crucial).

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Martin Schwichow
>Sent: Saturday, 02 October, 2021 12:00
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] how to interpreat a significant moderator variable but in-
>significant F test
>
>Hi,
>
>My name is Martin Schwichow and I am a Juniorprofessor for physics education at
>the University of Education Freiburg, Germany. Currently I am working on a meta-
>analysis on studies that teach the concept of floating and sinking to students. I
>am using the rma.mv function of the metaphor package.
>
>Everything works fine but for two moderator analyses, I get strange result. The
>omnibus F test is not significant but I have some significant moderator variables
>(see below).The variables represent dummy coded variables representing different
>levels of a factor. What does that mean and how can I interpret this results?
>Best,
>Martin
>
>Input: rma.mv(g, se2, mods=~theo_inq+theo_con+theo_cop, random = lev, tdist=TRUE,
>data=dat)
>
>Output
>Multivariate Meta-Analysis Model (k = 182; method: REML)
>
>Variance Components:
>
>            estim    sqrt  nlvls  fixed    factor
>sigma^2.1   0.0569  0.2386    182     no  g_number
>sigma^2.2   0.2529  0.5029     62     no  paper_id
>
>Test for Residual Heterogeneity:
>QE(df = 178) = 1423.5781, p-val < .0001
>
>Test of Moderators (coefficients 2:4):
>F(df1 = 3, df2 = 178) = 1.6910, p-val = 0.1706
>
>Model Results:
>
>          estimate      se     tval    pval    ci.lb    ci.ub
>intrcpt     1.0847  0.1558   6.9624  <.0001   0.7773   1.3921  ***
>theo_inq   -0.4419  0.2090  -2.1148  0.0358  -0.8543  -0.0296    *
>theo_con   -0.2019  0.1887  -1.0697  0.2862  -0.5742   0.1705
>theo_cop   -0.4507  0.3444  -1.3089  0.1922  -1.1303   0.2288
>
>--
>Jun.-Prof. Dr. Martin Schwichow
>Pädagogische Hochschule Freiburg · University of Education Freiburg
>Institut für Chemie, Physik, Technik und ihre Didaktiken
>Kunzenweg 21 · D-79117 Freiburg
>Tel: (+49) - (0)761 682 934
>
>www.scientific-reasoning.com