[R-meta] Question about Cohen's d and meta-regression

Thu Jan 4 10:57:16 CET 2018

Dear Angeline,

in addition to Wolfgang's excellent advice, here are two strategies that I
have employed successfully:

1) When interpreting d (or Hedges' g), try to convert it back to one of the
original metrics. I.e. if your original scale has a population standard
deviation of s in a typical sample, then  d*s is the shift in points on the
original scale. Depending on context, people find this useful frequently,
especially, when there is a dominant measure in the field.

2) I have used the strategy of Pigott & Hedges for power analysis in the
planning stage of meta-regressions. The book by Pigott is more accessible
than the paper and contains R code which is relatively easy to adapt.
Results will depend strongly on your assumptions, so I would caution you to
be rather conservative and assume smaller studies and larger random
effects. That said, 6 studies might or might not be sufficient.

Best wishes,
   Philipp

On Thu, Jan 4, 2018 at 10:24 AM, Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

> Dear Angeline,
>
> With respect to 1:
>
> This isn't part of your question, so feel free to ignore this, but I would
> be curious to see the details of how you are converting a within-subjects d
> to a between-subjects d (including the computation of the sampling
> variance).
>
> As for your actual question: In my opinion, neither U3, CLES, nor any
> other unitless measure will help with the interpretation of the actual
> magnitude of the effect. In his famous 1994 paper ("The earth is round (p <
> .05)"), Cohen himself also recommended moving away from 'standardized'
> measures of effect and instead advocated working with raw measures of
> effect. When meta-analyzing studies that used different measures to begin
> with, doing the meta-analysis with raw effects isn't possible, but we can
> still try to express the final result in terms of a raw effect. In
> particular, take the estimated (average) d value from the meta-analysis and
> multiply this with the SD of a scale that is well known/established in the
> area where the studies are coming from. For example, suppose a
> meta-analysis finds that children exposed to low levels of lead score lower
> on cognitive ability tests than non-exposed children with a d of -0.25.
> Then using a normed IQ test with an SD of 15, this implies a di
>  fference of -0.25 * 15 = -3.75 IQ points. I find this much easier to
> interpret than, for example: "The chances that a randomly drawn exposed
> child will score lower on a cognitive ablity test than a randomly drawn
> non-exposed child is 57%" (this would be the corresponding CLES result).
> The only difficulty with this approach might be that there isn't a normed
> measure with an 'established' SD. But one can often still identify a
> measure that is commonly used in a particular area, look at a bunch of
> studies to see what kind of SDs are typically observed, and then pick a
> sensible value.
>
> 2) Again, this is just my opinion, but such rules of thumb are usually
> useless. If I remember correctly, I've also read somewhere that one should
> have 5 (or 10) studies per moderator. I suspect these rules have arisen
> because analogous rules have been floating around for standard regression
> analyses (where they've also been criticized; for example: Maxwell, S. E.
> (2000). Sample size and multiple regression analysis. Psychological
> Methods, 5(4), 434-458.).
>
> Even with a relatively low value of k/p (k = number of studies, p = number
> of model coefficients), the Type I error rate for testing moderators is
> close to nominal as long as one employs something like the Knapp & Hartung
> correction. The problem is that power is likely to be very low then as
> well. One could try to do a proper power analysis (Hedges, L. V., & Pigott,
> T. D. (2004). The power of statistical tests for moderators in
> meta-analysis. Psychological Methods, 9(4), 426-445.), but I've never seen
> this done in practice (at least not reported). Something like a 'k per p'
> rule is of course much easier to implement.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-
> bounces at r-project.org] On Behalf Of Angeline Tsui
> Sent: Wednesday, 03 January, 2018 20:07
> To: r-sig-meta-analysis at r-project.org
> Subject: [R-meta] Question about Cohen's d and meta-regression
>
> Hello all,
>
> I have some questions about effect sizes and meta-regression.
>
> 1) I am using between-subject Cohen's d (but this effect size is converted
> from a within-subject d based on Morris and Deshan (2002) for my
> meta-analysis. The main reason of the conversion is to make this mean
> effect size comparable to other meta-analysis in my field, especially when
> some other meta-analysis may also report between-subject Cohen's d.
> However, I have concerns of interpreting the magnitude of the mean effect
> size.
>
> When I interpret the mean Cohen's d, I use the Cohen (1988) rule of thumb:
> 0.2 -> small, 0.5 -> medium and 0.8 -> large. However, a concern of
> applying this Cohen's values rule of thumb to the current meta-analysis is
> that it is not "context-specific". For example, a small effect size of 0.3
> can be regarded as large effect size in some other contexts.
>
> In this case, what other metrics would you recommend me to use to report
> the mean effect size magnitude? I have searched some resources online and
> see recommendation about the use of Cohen's U3 index or Common language
> effect size statistics (CLES)? From what I understand, both U3 index is a
> percentile index whereas CLES is a probability index. But I am not familiar
> with these two metrics, can someone give me recommendation which index is
> better?
>
> Finally, if I change my effect size to hedge's g because the sample size of
> each study is small and I need to correct it by converting Cohen's d to
> hedge's g. Can I still convert the mean hedge's g using the same formula of
> Cohen's U3 or CLES for interpreting the percentile/probability of the mean
> effect size?
>
> 2) I ran a meta-regression model with which include a number of moderators.
> My concern is the minimum sample size needed for each moderator level. I
> was not able to find resources for recommendation of sample size for
> moderator analysis. I did learn something from my teacher, she told me that
> the rule of thumb is at least 6 studies for a moderator analysis level
> (e.g., Gender is a moderator. Then I need to have at least 6 studies that
> examined male and at least 6 studies that examined female). Is this true? I
> again was not able to find this rule of thumb from books.
>
> Thanks,
> Angeline
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

-- 
Prof. Dr. Philipp Doebler
Technische Universität Dortmund
Fakultät Statistik
Vogelpothsweg 87
44227 Dortmund

Tel.: +49 231-755 8259
Fax: +49 231-755 3918
doebler at statistik.tu-dortmund.de
www.statistik.tu-dortmund.de/1261.html

Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie
ist ausschließlich für den Adressaten bestimmt. Sollten Sie nicht der
für diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den
Absender und vernichten Sie diese Mail. Vielen Dank.
Unbeschadet der Korrespondenz per E-Mail, sind unsere Erklärungen
ausschließlich final rechtsverbindlich, wenn sie in herkömmlicher
Schriftform (mit eigenhändiger Unterschrift) oder durch Übermittlung
eines solchen Schriftstücks per Telefax erfolgen.

Important note: The information included in this e-mail is confidential.
It is solely intended for the recipient. If you are not the intended
recipient of this e-mail please contact the sender and delete this
message. Thank you.
Without prejudice of e-mail correspondence, our statements are only
legally binding when they are made in the conventional written form
(with personal signature) or when such documents are sent by fax.

	[[alternative HTML version deleted]]