[R-meta] Question about Cohen's d and meta-regression

[Viechtbauer Wolfgang (SP)]

Dear Angeline,

With respect to 1:

This isn't part of your question, so feel free to ignore this, but I would be curious to see the details of how you are converting a within-subjects d to a between-subjects d (including the computation of the sampling variance).

As for your actual question: In my opinion, neither U3, CLES, nor any other unitless measure will help with the interpretation of the actual magnitude of the effect. In his famous 1994 paper ("The earth is round (p < .05)"), Cohen himself also recommended moving away from 'standardized' measures of effect and instead advocated working with raw measures of effect. When meta-analyzing studies that used different measures to begin with, doing the meta-analysis with raw effects isn't possible, but we can still try to express the final result in terms of a raw effect. In particular, take the estimated (average) d value from the meta-analysis and multiply this with the SD of a scale that is well known/established in the area where the studies are coming from. For example, suppose a meta-analysis finds that children exposed to low levels of lead score lower on cognitive ability tests than non-exposed children with a d of -0.25. Then using a normed IQ test with an SD of 15, this implies a difference of -0.25 * 15 = -3.75 IQ points. I find this much easier to interpret than, for example: "The chances that a randomly drawn exposed child will score lower on a cognitive ablity test than a randomly drawn non-exposed child is 57%" (this would be the corresponding CLES result). The only difficulty with this approach might be that there isn't a normed measure with an 'established' SD. But one can often still identify a measure that is commonly used in a particular area, look at a bunch of studies to see what kind of SDs are typically observed, and then pick a sensible value.

2) Again, this is just my opinion, but such rules of thumb are usually useless. If I remember correctly, I've also read somewhere that one should have 5 (or 10) studies per moderator. I suspect these rules have arisen because analogous rules have been floating around for standard regression analyses (where they've also been criticized; for example: Maxwell, S. E. (2000). Sample size and multiple regression analysis. Psychological Methods, 5(4), 434-458.).

Even with a relatively low value of k/p (k = number of studies, p = number of model coefficients), the Type I error rate for testing moderators is close to nominal as long as one employs something like the Knapp & Hartung correction. The problem is that power is likely to be very low then as well. One could try to do a proper power analysis (Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 426-445.), but I've never seen this done in practice (at least not reported). Something like a 'k per p' rule is of course much easier to implement.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Angeline Tsui
Sent: Wednesday, 03 January, 2018 20:07
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Question about Cohen's d and meta-regression

Hello all,

I have some questions about effect sizes and meta-regression.

1) I am using between-subject Cohen's d (but this effect size is converted
from a within-subject d based on Morris and Deshan (2002) for my
meta-analysis. The main reason of the conversion is to make this mean
effect size comparable to other meta-analysis in my field, especially when
some other meta-analysis may also report between-subject Cohen's d.
However, I have concerns of interpreting the magnitude of the mean effect
size.

When I interpret the mean Cohen's d, I use the Cohen (1988) rule of thumb:
0.2 -> small, 0.5 -> medium and 0.8 -> large. However, a concern of
applying this Cohen's values rule of thumb to the current meta-analysis is
that it is not "context-specific". For example, a small effect size of 0.3
can be regarded as large effect size in some other contexts.

In this case, what other metrics would you recommend me to use to report
the mean effect size magnitude? I have searched some resources online and
see recommendation about the use of Cohen's U3 index or Common language
effect size statistics (CLES)? From what I understand, both U3 index is a
percentile index whereas CLES is a probability index. But I am not familiar
with these two metrics, can someone give me recommendation which index is
better?

Finally, if I change my effect size to hedge's g because the sample size of
each study is small and I need to correct it by converting Cohen's d to
hedge's g. Can I still convert the mean hedge's g using the same formula of
Cohen's U3 or CLES for interpreting the percentile/probability of the mean
effect size?

2) I ran a meta-regression model with which include a number of moderators.
My concern is the minimum sample size needed for each moderator level. I
was not able to find resources for recommendation of sample size for
moderator analysis. I did learn something from my teacher, she told me that
the rule of thumb is at least 6 studies for a moderator analysis level
(e.g., Gender is a moderator. Then I need to have at least 6 studies that
examined male and at least 6 studies that examined female). Is this true? I
again was not able to find this rule of thumb from books.

Thanks,
Angeline