# [R-meta] Effect sizes for mixed-effects models

James Pustejovsky jepu@to @end|ng |rom gm@||@com
Tue Oct 22 05:12:04 CEST 2019

```Lena,

The formula you tried from Hedges 2007 is derived under the assumption that
treatment assignment is at the cluster level, so I don't think it will work
for your mixed design. The following post might be useful to answer your
questions:
https://www.jepusto.com/alternative-formulas-for-the-smd/
In it, I suggest a quite general approach to estimating the variance of a
standardized mean difference effect size, even if it is based on a complex
experimental design. Suppose that you calculate the SMD estimate as

d = b / S,

where b is the unstandardized mean difference (which in your design
involves a combination of within- and between-Ss comparisons) and S is the
standard deviation of the outcome, which generally might involve a sum of
multiple variance components. A delta-method approximation to the variance
of d is

Vd = (SEb / S)^2 + d^2 / (2 v),

where SEb is the standard error of b, S is the denominator of the effect
size estimate, d is the effect size estimate, and v is the degrees of
freedom of S^2, defined by v = 2[ E(S^2)]^2 / Var(S^2). The SEb should
usually be reported in primary studies (or can be back-calculated from t
statistics or CIs). Thus, the only tricky bit is to find the degrees of
freedom for the standardizing variance S^2. You might need to just make a
rough approximation, based on for instance the total number of
participants. Using a rough approximation (e.g., v = 30) should not have
much effect on the total estimated variance Vd unless d is very large, so
personally I would not worry too much about getting it perfect.

As I explain in the post, you can also use the degrees of freedom v to do
Hedges' g correction, taking

g = J(v) * d,

where J(v) = 1 - 3 / (4 * v - 1). Again, it's not worth worrying about
getting the degrees of freedom perfect. Consider that J(30) = 0.9748 and
J(60) = 0.9874, so the g estimate will differ by only a tiny amount
depending on the degrees of freedom you use.

James

On Sat, Oct 19, 2019 at 2:41 PM Lena Schäfer <lenaschaefer2304 using gmail.com>
wrote:

> Hi everyone,
>
> I am writing to ask two questions related to the calculation of effect
> sizes for mixed-effects models for a meta-analysis.
> To derive effect sizes for mixed-effects models, we generally follow the
> Hedges 2007 paper (
> https://journals.sagepub.com/doi/abs/10.3102/1076998606298043?journalCode=jebb
> <
> https://journals.sagepub.com/doi/abs/10.3102/1076998606298043?journalCode=jebb>)
> and a blogpost by Jake Westfall on effect-size calculations for
> within-subjects designs (
> http://jakewestfall.org/blog/index.php/2016/03/25/five-different-cohens-d-statistics-for-within-subject-designs/
> <
> http://jakewestfall.org/blog/index.php/2016/03/25/five-different-cohens-d-statistics-for-within-subject-designs/
> >):
>
> 1.     Variance for complex mixed-effects models
> While the calculation of Cohen’s d is unproblematic (formula 8 on page 346
> in Hedges, 2007), the calculation of the respective variance turned out to
> be difficult for complex study designs. Hedge’s provided the following
> formula () to derive V(dw):
>
> V(dw) = ((NT + NC) / (NT * NC)) * ((1+(n-1)p)/(1-p)) + ((dw^2) / (2(N –
> M)))
>
> with NT referring to the number of observations in the treatment group, NC
> referring to the number of observations in the control group, N referring
> to the total number of observations (NT + NC  = N), n referring to the
> number of observations per cluster, p referring to the ICC, and M referring
> to the number of clusters.
> For our meta-analysis, we want to derive the variance related to Cohen’s d
> for a mixed-subjects design with some participant conducting a task only in
> the control condition and other participants conducting the task in the
> control and in the experimental condition (within-subjects design). Since
> the number of observations per cluster differs (some participants have 30
> observations, others have 60) we decided to use the variance formula for
> unequal cluster sample sizes in which n is substituted with the cluster
> sample size ñ (formula 18 on page 350):
>
> ñ = ((NC * ΣmTi = 1 (nTi)^2) / (NT * N)) +  ((NT * ΣmCi = 1 (nCi)^2) / (NC
> * N))
>
> iWhile we expected that this formula would yield an unequal cluster sample
> size between 30 and 60, it gives us a value of 30 (which is equal to the
> cluster sample size if this would be a between-subjects design). This
> suggests that the formula cannot account for the participants which are
> both in the control and the experimental condition. Do you have any advice
> on how we could derive an accurate variance estimate for such a design?
>
> 2. Turning Cohen’s d into Hedge’s g for mixed-models
> Finally, we want to transform Cohen’s d into Hedge’s g using:
>
> g(d) = d * (1- ((3) / (4 * df - 1))
>
> We are uncertain how to best estimate the dfs in our mixed-models. We
> considered using Kenward-Roger approximated dfs but this does not seem
> feasible since we only have access to parts of the raw data-sets used to
> derive dw and V{dw}. Potentially, another option would be to estimate the
> dfs via the effective sample size. This seems more feasible since the
> authors of primary papers provided us with the ICC related to each model.
>
> If you have any thoughts on this, we would greatly appreciate it if you
> could let us know what you think. Thank you for taking the time to consider
> our request, and please don’t hesitate to reach out if anything is unclear.
>
> Thank you very much and best regards,
> Lena Schäfer
>
> On behalf of a collaborative team that additionally includes Leah
> Somerville (head of the Affective Neuroscience and Development Laboratory),
> Katherine Powers (former postdoc in the Affective Neuroscience and
> Development Laboratory) and Bernd Figner (Radboud University).
>
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

[[alternative HTML version deleted]]

```