[R-meta] Question about Cohen's d and meta-regression

[Angeline Tsui]

Dear Wolfgang and Philipp,

Thank you so much for the detailed response. This is exactly what I am
looking for. I will look into the Knapp & Hartung correction and also do a
power analysis of meta-regression. Thanks again for the recommended
resources.

Re Wolfgang as for the question about changing within-subject Cohen's d to
between-subject Cohen's d: To be honest, I have struggled quite a lot
before making this decision. The primary reason is due to recommendations
from researchers in my field. They once advocated meta-analysis in my field
to change within-subject Cohen's d to between-subject Cohen's d. I think it
is simply related to converting meta-analysis mean effect size to a common
metric, so that we can directly compare mean effect sizes across
meta-analyses. Another reason (I think I may be wrong and please feel free
to correct me) is that I was worried if I can still apply the Cohen's rule
of thumb (0.2, 0.5, 0.8) to interpret within-subject Cohen's d. I know that
this rule of thumb applies to between-subject Cohen's d, but I really am
not sure if we can do this for within-subject Cohen's d. Specifically, the
standard deviations are different between within-subject and
between-subject designs. The standard deviations for within-subject design
are theoretically smaller since we are controlling for individual
differences. Thus, I changed my metric to within-subject Cohen's d just for
the ease of interpretation as well.

Best,
Angeline

On Thu, Jan 4, 2018 at 4:57 AM, Philipp Doebler <
doebler at statistik.tu-dortmund.de> wrote:

> Dear Angeline,
>
> in addition to Wolfgang's excellent advice, here are two strategies that I
> have employed successfully:
>
> 1) When interpreting d (or Hedges' g), try to convert it back to one of
> the original metrics. I.e. if your original scale has a population standard
> deviation of s in a typical sample, then  d*s is the shift in points on the
> original scale. Depending on context, people find this useful frequently,
> especially, when there is a dominant measure in the field.
>
> 2) I have used the strategy of Pigott & Hedges for power analysis in the
> planning stage of meta-regressions. The book by Pigott is more accessible
> than the paper and contains R code which is relatively easy to adapt.
> Results will depend strongly on your assumptions, so I would caution you to
> be rather conservative and assume smaller studies and larger random
> effects. That said, 6 studies might or might not be sufficient.
>
> Best wishes,
>    Philipp
>
> On Thu, Jan 4, 2018 at 10:24 AM, Viechtbauer Wolfgang (SP) <
> wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
>
>> Dear Angeline,
>>
>> With respect to 1:
>>
>> This isn't part of your question, so feel free to ignore this, but I
>> would be curious to see the details of how you are converting a
>> within-subjects d to a between-subjects d (including the computation of the
>> sampling variance).
>>
>> As for your actual question: In my opinion, neither U3, CLES, nor any
>> other unitless measure will help with the interpretation of the actual
>> magnitude of the effect. In his famous 1994 paper ("The earth is round (p <
>> .05)"), Cohen himself also recommended moving away from 'standardized'
>> measures of effect and instead advocated working with raw measures of
>> effect. When meta-analyzing studies that used different measures to begin
>> with, doing the meta-analysis with raw effects isn't possible, but we can
>> still try to express the final result in terms of a raw effect. In
>> particular, take the estimated (average) d value from the meta-analysis and
>> multiply this with the SD of a scale that is well known/established in the
>> area where the studies are coming from. For example, suppose a
>> meta-analysis finds that children exposed to low levels of lead score lower
>> on cognitive ability tests than non-exposed children with a d of -0.25.
>> Then using a normed IQ test with an SD of 15, this implies a di
>>  fference of -0.25 * 15 = -3.75 IQ points. I find this much easier to
>> interpret than, for example: "The chances that a randomly drawn exposed
>> child will score lower on a cognitive ablity test than a randomly drawn
>> non-exposed child is 57%" (this would be the corresponding CLES result).
>> The only difficulty with this approach might be that there isn't a normed
>> measure with an 'established' SD. But one can often still identify a
>> measure that is commonly used in a particular area, look at a bunch of
>> studies to see what kind of SDs are typically observed, and then pick a
>> sensible value.
>>
>> 2) Again, this is just my opinion, but such rules of thumb are usually
>> useless. If I remember correctly, I've also read somewhere that one should
>> have 5 (or 10) studies per moderator. I suspect these rules have arisen
>> because analogous rules have been floating around for standard regression
>> analyses (where they've also been criticized; for example: Maxwell, S. E.
>> (2000). Sample size and multiple regression analysis. Psychological
>> Methods, 5(4), 434-458.).
>>
>> Even with a relatively low value of k/p (k = number of studies, p =
>> number of model coefficients), the Type I error rate for testing moderators
>> is close to nominal as long as one employs something like the Knapp &
>> Hartung correction. The problem is that power is likely to be very low then
>> as well. One could try to do a proper power analysis (Hedges, L. V., &
>> Pigott, T. D. (2004). The power of statistical tests for moderators in
>> meta-analysis. Psychological Methods, 9(4), 426-445.), but I've never seen
>> this done in practice (at least not reported). Something like a 'k per p'
>> rule is of course much easier to implement.
>>
>> Best,
>> Wolfgang
>>
>> -----Original Message-----
>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bo
>> unces at r-project.org] On Behalf Of Angeline Tsui
>> Sent: Wednesday, 03 January, 2018 20:07
>> To: r-sig-meta-analysis at r-project.org
>> Subject: [R-meta] Question about Cohen's d and meta-regression
>>
>> Hello all,
>>
>> I have some questions about effect sizes and meta-regression.
>>
>> 1) I am using between-subject Cohen's d (but this effect size is converted
>> from a within-subject d based on Morris and Deshan (2002) for my
>> meta-analysis. The main reason of the conversion is to make this mean
>> effect size comparable to other meta-analysis in my field, especially when
>> some other meta-analysis may also report between-subject Cohen's d.
>> However, I have concerns of interpreting the magnitude of the mean effect
>> size.
>>
>> When I interpret the mean Cohen's d, I use the Cohen (1988) rule of thumb:
>> 0.2 -> small, 0.5 -> medium and 0.8 -> large. However, a concern of
>> applying this Cohen's values rule of thumb to the current meta-analysis is
>> that it is not "context-specific". For example, a small effect size of 0.3
>> can be regarded as large effect size in some other contexts.
>>
>> In this case, what other metrics would you recommend me to use to report
>> the mean effect size magnitude? I have searched some resources online and
>> see recommendation about the use of Cohen's U3 index or Common language
>> effect size statistics (CLES)? From what I understand, both U3 index is a
>> percentile index whereas CLES is a probability index. But I am not
>> familiar
>> with these two metrics, can someone give me recommendation which index is
>> better?
>>
>> Finally, if I change my effect size to hedge's g because the sample size
>> of
>> each study is small and I need to correct it by converting Cohen's d to
>> hedge's g. Can I still convert the mean hedge's g using the same formula
>> of
>> Cohen's U3 or CLES for interpreting the percentile/probability of the mean
>> effect size?
>>
>> 2) I ran a meta-regression model with which include a number of
>> moderators.
>> My concern is the minimum sample size needed for each moderator level. I
>> was not able to find resources for recommendation of sample size for
>> moderator analysis. I did learn something from my teacher, she told me
>> that
>> the rule of thumb is at least 6 studies for a moderator analysis level
>> (e.g., Gender is a moderator. Then I need to have at least 6 studies that
>> examined male and at least 6 studies that examined female). Is this true?
>> I
>> again was not able to find this rule of thumb from books.
>>
>> Thanks,
>> Angeline
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>
>
>
> --
> Prof. Dr. Philipp Doebler
> Technische Universität Dortmund
> Fakultät Statistik
> Vogelpothsweg 87
> <https://maps.google.com/?q=Vogelpothsweg+87+44227+Dortmund&entry=gmail&source=g>
> 44227 Dortmund
> <https://maps.google.com/?q=Vogelpothsweg+87+44227+Dortmund&entry=gmail&source=g>
>
> Tel.: +49 231-755 8259 <+49%20231%207558259>
> Fax: +49 231-755 3918 <+49%20231%207553918>
> doebler at statistik.tu-dortmund.de
> www.statistik.tu-dortmund.de/1261.html
>
>
> Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie
> ist ausschließlich für den Adressaten bestimmt. Sollten Sie nicht der
> für diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den
> Absender und vernichten Sie diese Mail. Vielen Dank.
> Unbeschadet der Korrespondenz per E-Mail, sind unsere Erklärungen
> ausschließlich final rechtsverbindlich, wenn sie in herkömmlicher
> Schriftform (mit eigenhändiger Unterschrift) oder durch Übermittlung
> eines solchen Schriftstücks per Telefax erfolgen.
>
> Important note: The information included in this e-mail is confidential.
> It is solely intended for the recipient. If you are not the intended
> recipient of this e-mail please contact the sender and delete this
> message. Thank you.
> Without prejudice of e-mail correspondence, our statements are only
> legally binding when they are made in the conventional written form
> (with personal signature) or when such documents are sent by fax.
>



-- 
Best Regards,
Angeline

	[[alternative HTML version deleted]]