[R-meta] Negative r value for effect sizes within studies

[Reza Norouzian]

I should add that it is entirely "possible" to encounter outcomes that
are measured in opposite directions across studies and that can be
meaningfully part of the same multivariate meta-analysis. But like I
mentioned, syncing the outcomes' directions often resolves the issue.

Here is a substantive example.

Studies that research the efficacy of teacher's feedback on students'
(English Language Learners) writing competence often focus on various
related linguistic features (e.g., verbs, articles etc.). However, due
to the debates over how these linguistic features ought to be
measured, these studies tend to measure some linguistic features "by
tradition" using the number of errors, but others using the number of
correct usage (I'm simplifying the actual measures to make this
discussion more general, but the directions of outcomes run opposite
to one another for different linguistic features).

The fact that these outcome levels are related is of great interest,
hence motivating a multivariate meta-analysis. On the other hand,
almost always these studies are multi-group to compare different types
of feedback and longitudinal to measure the durability of feedback's
effect on developing students' writing competence (to make matters
even more complex, they even use more than one comparison/control
group).

In a situation like what I described above (and more generally as
described in my previous email), if not synced, these opposing
outcomes could create a difficult situation where each study could
have a combination of positively and negatively correlated estimates
of effect size in it.

So, synching the outcome directions, at least for me, is inevitable.

Reza

On Fri, Sep 3, 2021 at 12:19 PM James Pustejovsky <jepusto using gmail.com> wrote:
>
> To add to Wolfgang's comments, yes it is possible in principle. However, on
> a practical level, it seems like it would be a very odd situation where you
> would include two outcomes in the same meta-analysis that are *negatively*
> correlated with each other. Say that one intervention study reports a
> negative effect size estimate for depression and a positive effect size for
> positive affect, two outcomes which are negatively correlated. It would not
> make much practical sense to average the negative ES for depression and the
> positive ES for positive affect together, which suggests that it would be
> very odd to include both in a single meta-analysis.
>
> On Fri, Sep 3, 2021 at 12:09 PM Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> > Dear Stefanou,
> >
> > Would be unusual, but in principle, yes, it can. However, only certain
> > values of r are actually possible. Let's say there is a study with 5
> > estimates and you assume r = -0.6. Then the correlation matrix for this
> > study would be:
> >
> > v <- matrix(-.6, 5, 5); diag(v) <- 1
> >
> > which is not positive definite, which we can see by examining its
> > eigenvalues:
> >
> > round(eigen(v)$values, 4)
> >
> > (note the last eigenvalue is negative). The largest negative correlation
> > possible when p=5 (to get at least a positive semi-definite matrix) is
> > -0.25:
> >
> > v <- matrix(-.25, 5, 5); diag(v) <- 1
> > round(eigen(v)$values, 4)
> >
> > (the last eigenvalue is now essentially 0).
> >
> > More generally, the lower bound is:
> >
> > -1/(p-1)
> >
> > So, you need to determine which study has the largest number of estimates
> > and based on that, you know what the lower bound for r is (at least, when
> > we assume that the correlation matrix is compound symmetric - as above).
> >
> > Best,
> > Wolfgang
> >
> > >-----Original Message-----
> > >From: R-sig-meta-analysis [mailto:
> > r-sig-meta-analysis-bounces using r-project.org] On
> > >Behalf Of Stefanou Revesz
> > >Sent: Friday, 03 September, 2021 18:47
> > >To: R meta
> > >Subject: [R-meta] Negative r value for effect sizes within studies
> > >
> > >Dear All,
> > >
> > >This may be too basic to ask. But can the r value chosen to represent
> > >the common correlation among effect sizes within studies be a negative
> > >one?
> > >
> > >For example, is there a situation where I would need to use the following?
> > >
> > >impute_covariance_matrix(..., r = -.6)
> > >
> > >Thank you,
> > >Stefanou
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis using r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis