[R-meta] About how to interpret the following P-value from the result

[Michael Dewey]

Dear Nick

The omnibus test for moderators has already told you that the hypothesis 
that they are all equal cannot be rejected. If you go on to try to test 
combinations of them for equality (which could include testing pairwise 
differences) then you are likely to capitalise on chance and your 
p-values and confidence intervals will not have their usual meaning.

Having said that if you really, really want to do it look at the btt 
parameter.

Michael

On 28/10/2023 15:48, Nick Chen via R-sig-meta-analysis wrote:
> Dear Micheal,
> 
>       Thank you for the clarification. This has bugged me for quite a long
> time. Then one follow-up question, is there any test that can be done to
> show which of the element in the subgroup (here in this case, intercpt,
> DeviceMD, DeviceN/A, DeviceV/A) has significant difference than the others?
> 
> Nick
> 
> Michael Dewey via R-sig-meta-analysis <r-sig-meta-analysis using r-project.org>於
> 2023年10月28日 週六，下午8:38寫道：
> 
>> Dear Nick
>>
>> Comment in-line
>>
>> On 28/10/2023 10:08, 英文科陳品誠 via R-sig-meta-analysis wrote:
>>> I'm currently performing a meta-analysis on the relationship between
>>> "Digital game-based learning" and "English vocabulary acquisition". And
>> I'm
>>> using the random effect model for moderator analysis since the
>>> heterogeneity effect is quite high. The following is the code I use for
>>> performing the moderator analysis for one moderator in my dataset,
>> "device
>>> used":
>>>
>>>> mod.Device <- rma.mv(yi = y, V = v,+                        slab =
>> studyID, data = DG,+                        random = ~ 1 |
>> studyID/effectsizeID,+                        test = "t", method =
>> "REML",+                        mods = ~ Device)> mod.Device
>>>
>>> and here is the result:
>>>
>>>
>>>
>>> Multivariate Meta-Analysis Model (k = 29; method: REML)
>>>
>>> Variance Components:
>>>
>>>               estim    sqrt  nlvls  fixed                factor
>>> sigma^2.1  0.0663  0.2575     28     no               studyID
>>> sigma^2.2  0.0000  0.0000     29     no  studyID/effectsizeID
>>>
>>> Test for Residual Heterogeneity:
>>> QE(df = 25) = 44.8744, p-val = 0.0086
>>>
>>> Test of Moderators (coefficients 2:4):
>>> F(df1 = 3, df2 = 25) = 0.8975, p-val = 0.4563
>>>
>>> Model Results:
>>>
>>>              estimate      se     tval  df    pval    ci.lb   ci.ub
>>> intrcpt      0.7746  0.1008   7.6856  25  <.0001   0.5670  0.9822  ***
>>> DeviceMD     0.0898  0.1681   0.5343  25  0.5978  -0.2564  0.4361
>>> DeviceN/A   -0.2275  0.2338  -0.9732  25  0.3398  -0.7091  0.2540
>>> DeviceV/A    0.2968  0.2936   1.0107  25  0.3219  -0.3080  0.9015
>>>
>>> intrcpt    = computer
>>>
>>> DeviceMD   = mobile devices
>>>
>>> DeviceN/A  = not available
>>>
>>> DeviceV/A  = VR or AR devices
>>> ---
>>>
>>> and this is what I'm a little bit confused on. Can anyone help me clarify
>>> whether my explanation is correct or wrong?
>>> (1) the p-value under "*Test of moderators*" which is 0.4563, does this
>>> mean that this moderator (device used) does not have a significant impact
>>> on English vocabulary learning using digital game-based learning? (Since
>>> the p-value here is larger than 0.05)
>>
>> Yes, that is what it implies.
>>
>>> (2) the p-value of "*intrcpt*" which is <.0001, does this mean that using
>>> computers to learn English vocabulary still has a significant difference
>>> among the four types of devices?
>>>
>>
>> No, that is the estimated value of the vocabulary score for the fourth
>> category of your moderator variable. It is unlikely that it has any
>> scientific value.
>>
>> Michael
>>
>>> 陳品誠 (Nick Chen)
>>> Email: t571 using wlgsh.tp.edu.tw <t5741 using wlgsh.tp.edu.tw>
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>> To manage your subscription to this mailing list, go to:
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>
>> --
>> Michael
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 

-- 
Michael