[R-meta] Multivariate meta-analysis - moderator analysis and tau squared

[James Pustejovsky]

Hi Mika,

Comments below.

James

On Thu, Sep 24, 2020 at 12:45 PM Mika Manninen <mixu89 using gmail.com> wrote:


> #Here I sometimes get the following warning message: "V appears to be not
> positive definite". 1. Should this be ignored?
>

This means that one of your input correlation matrices is not positive
definite, i.e., it is not a valid correlation matrix. It could be due to a
typo or rounding of the entries. You can find the offending matrix using
the following

isPosDef <- function(x) all(eigen(x)$values > 0)
sapply(corlist, isPosDef)

Although it's probably a minor issue, I would still suggest correcting
before proceeding.

#Each level of motivation has its own variance component in the output
> (tau). 2. How could I obtain the Q-values (and consequently I squared) to
> report?
>
>
I would recommend reporting the tau estimates directly. Because they are
model parameters, they are more meaningful and more directly interpretable
than Q or I-squared.

As far as Q, there are a number of different ways to define Q in
multivariate models (or generally, models with multiple variance
components), some of which are global rather than being specific to the
dimension of the outcome. Do you want to report Q as a global description
of excess heterogeneity, or do you want something specific to the outcome
dimension?


> #If I now wanted to do a moderator analysis with each level of motivation
> using the categorical variable "setting"
> #to see if there is significant difference between the two settings for
> motivation 1,2,3 etc. 3. Would the following be correct?
>
> res <-  rma.mv(g, V, mods = ~ factor(motivation)+
> factor(motivation):I(setting) -1, random = ~ factor(motivation) | study,
> struct="UN", data=meta)
> res
>
> res <- rma.mv(g, V, mods = ~ factor(motivation)*I(setting) -1, random = ~
> factor(motivation) | study, struct="UN", data=meta)
> res
>
> #I am not exactly sure what I should be interpreting. I would like to know
> if setting moderates the effect of the intervention on motivation levels
> 1,2,3,4,5,6 (or can I do that - is the comparison done "overall"? )
>
> Is your question about setting moderates for *any* motivation level? Or
whether it moderates for  *specific* motivation levels?

If it is the former, then you can do a likelihood ratio test comparing the
model with moderator to the model without. You would, however, have to
switch to using ML rather than REML estimation for the variance components.
Syntax as follows:

res_ML <- rma.mv(g, V, mods = ~ factor(motivation) - 1,
                 random = ~ factor(motivation) | study,
                 struct="UN", data = meta,
                 method = "ML")
mod_ML <- update(res_ML, mods = ~ factor(motivation)*I(setting) -1)
anova(res_ML, mod_ML)

If it is the latter, then the first set of syntax gives you coefficient
estimates for the difference in average effect size between setting = 1
versus setting = 0, for each distinct level of motivation, so you can
interpret those coefficients (CIs, t-tests) directly. If you're looking at
the t-tests for all six levels, it would be prudent to use a correction for
multiple comparisons.


> #Same goes for the Egger's type test, 4. how should the results of the
> moderator analyses be interpreted if I were to use the variance estimate or
> its inverse (v) as the predictor to test for publication bias/funnel plot
> asymmetry
>

If these effect sizes are standardized mean differences, then you'll need
to use a modified measure of precision rather than the variance or standard
error of the effect sizes. Details here:
https://www.jepusto.com/publication/testing-for-funnel-plot-asymmetry-of-smds/

There are at least two ways to implement Egger's test in this setting. One
would be to simply add the modified measure of precision as a predictor. A
significant slope coefficient would be indicative of small-study effects.
Alternately, you could interact the predictor with the levels of motivation
and then report the likelihood ratio test, as with the previous question.
It is hard to say which approach is more powerful generally. Perhaps others
on the listserv have insights.


> #5. Any other advice to report the results (e.g., how to display the forest
> plot and is there any sense to do a funnel plot?)
>

You could make a forest plot with the points and whiskers in different
color shades corresponding to the levels of motivation. Alternately, make
separate forest plots per level of motivation. Similarly, a funnel plot
with points in different colors corresponding to the levels of motiviation,
or make separate funnels per level of motivation.

	[[alternative HTML version deleted]]