[R-meta] Questions about multilevel meta-analysis structure

[James Pustejovsky]

Hi Isaac,

Responses below.

Best,
James


On Mon, Oct 23, 2023 at 10:10 PM Isaac Calvin Saywell <
isaac.saywell using adelaide.edu.au> wrote:

> <snip>
>
> (1) Plotting the moderating influence of CR indicator (CRQ scores,
> Education level, Education years etc) on the main effect of CR resulted in
> the following:
> https://universityofadelaide.box.com/s/7r3km1oxdq8l0k8v42o3y7jtblq2ctel
>
> As you can see, the estimated central tendency for CRQ scores is very
> inflated relative to the individual effect sizes (it is also somewhat
> inflated for the other CR indicators). We are assuming the estimate for CRQ
> scores is inflated because effect sizes were derived from only one study?
> If this is correct, should it be dropped from the moderation analysis? If
> not, do you have any ideas as to why this occurred? (Several of our other
> moderation analyses are showing the same problem, though not to the same
> extent as this one).
>

This is indeed pretty weird. I'm speculating a bit, but my guess is that it
happens because of the confluence of several conditions:
1) There is only one study (call it study X) with effect sizes for CRQ and
that study also includes other types of CR indicators.
2) You're using a multivariate model that "borrows" information across
categories.
3) The effect sizes for other CR indicators in study X are below average
relative to the overall effect size for that type of indicator. Because the
model borrows information across categories, it infers that the observed
effect sizes for CRQ from study X are also a bit below average relative to
the overall effect size for CRQ. It therefore estimates the overall average
to be above the mean of all the observed CRQ effects.

This sort of reasoning is valid---and the prediction is reasonable, I would
argue---if the working model that you're using is a good description of the
real data generating process. But if it's haphazard or mis-specified, then
the model could generate some pretty lousy predictions. Given that you've
only got one study with effect sizes in this category, you have very little
information to evaluate the modeling assumptions regarding how CRQ scores
are related to other types of CR indicators, so pretty much any modeling
assumption you make will be difficult to evaluate. Dropping effects based
on CRQ scores (or perhaps just reporting them separately) might be prudent,
but this sort of judgement really depends on knowing the specific context
of your study.


> (2) In regards to the calculations used to compute V via the
> impute_covariance_matrix() function, would it be preferential to manually
> calculate this from the raw data? If so, are you able to provide resources
> on how to do this?
>

I don't know of any work that provides the covariances between semipartial
correlations (maybe Ariel or Betsy know?). It could be derived (
https://www.jepusto.com/multivariate-delta-method/) but doing so would be
pretty tedious.

Alternately, you could use seemingly unrelated regression. Say you have a
dataset with outcomes Y1, Y2, Y3,..., CR predictors X1, X2, X3,..., and one
common predictor U. Re-arrange the data so that you have a row for every
observation and every unique combination of a Y and an X. The long-form
data would need to look like this:
ID Yid Xid Y  X     U
1  1     1   y1  x1  u1
1  2     1   y2  x1  u1
1  3     1   y3  x1  u1
1  1     2   y1  x2  u1
1  2     2   y2  x2  u1
1  3     2   y3  x2  u1
1  1     3   y1  x3  u1
1  2     3   y2  x3  u1
1  3     3   y3  x3  u1
1  1     4   y1  x4  u1
1  2     4   y2  x4  u1
1  3     4   y3  x4  u1
2  1     1   y1  x1  u2
2  2     1   y2  x1  u2
2  3     1   y3  x1  u2
2  1     2   y1  x2  u2
2  2     2   y2  x2  u2
2  3     2   y3  x2  u2
2  1     3   y1  x3  u2
2  2     3   y2  x3  u2
2  3     3   y3  x3  u2
2  1     4   y1  x4  u2
2  2     4   y2  x4  u2
2  3     4   y3  x4  u2
3  1     1   y1  x1  u3
3  2     1   y2  x1  u3
3  3     1   y3  x1  u3
3  1     2   y1  x2  u3
3  2     2   y2  x2  u3
3  3     2   y3  x2  u3
3  1     3   y1  x3  u3
3  2     3   y2  x3  u3
3  3     3   y3  x3  u3
3  1     4   y1  x4  u3
3  2     4   y2  x4  u3
3  3     4   y3  x4  u3
etc.
You can then fit the regression model
Y = 0 + factor(Yid) * factor(Xid) + factor(Yid) * factor(Xid):(X + U + X *
X)
This should give you something that's exactly equivalent to fitting the
basic model Y = X + U + X * U for each unique combination of Yid and Xid,
but it puts all the coefficients together in one model. Then compute
cluster-robust standard errors, clustering by ID variable to account for
the fact that you've got the same value of Y or X repeated across several
rows. Take the sub-matrix of the variance-covariance matrix corresponding
to the beta coefficients of interest.

Another alternative would be to just bootstrap the sampling variance
covariance matrix. For a single primary study, this process involves the
following:
1. Draw a bootstrap re-sample of observations from the original data.
2. Compute all the semi-partial correlation coefficients of interest. Store
these "bootstrap r_sp" values in a vector.
3. Repeat steps 1 and 2 a bunch of time (500-1000 times).
4. Calculate the sample variance-covariance matrix of the bootstrap r_sp
values. Use this as the V matrix for that primary study in the
meta-analysis.


> (3) Finally, given the unique nature of our meta-analysis and the large
> number of effect sizes that are derived from often very similar regression
> models, are there any problems you can see regarding our use of these
> functions?
>

I would say that your application of multilevel meta-analysis is maybe a
little bit unusual since you've got a relatively modest number studies and
a whole lot of effect sizes that capture slightly different
operationalizations of the constructs of interest. I don't know anything
about the substantive area you're looking at or about the properties of
different measures of cognitive reserve, but I could imagine someone
objecting to taking such a "kitchen sink" approach to your research aims,
raising questions such as:  Are some CR measures better than others? Why
treat all different cognitive outcomes as more or less equal or
interchangeable?

One last thought. You might find the following article
interesting/stimulating:
Fernández-Castilla, B., Maes, M., Declercq, L. *et al.* A demonstration and
evaluation of the use of cross-classified random-effects models for
meta-analysis. *Behav Res *51, 1286–1304 (2019).
https://doi.org/10.3758/s13428-018-1063-2
One of the motivating use-cases they describe is a situation similar to
yours, where you have a variety of different operationalizations of a
construct and the idea is to treat the specific operationalization as a
random effect.

	[[alternative HTML version deleted]]