[R-meta] Question Re: Correcting for Statistical Artifacts

[Reza Norouzian]

Tori,

You may want to double-check with the psychmeta package's main author
(Brenton Wiernik), but the methods as implemented in the psychmeta
package don't seem to handle (multivariate) multilevel meta-regression
models which broadly incorporate nested models that you're referring
to.

In addition, the well-known sources of the biases (e.g., measurement
error, selection bias, ...) can potentially, if not commonly, occur
together in the primary studies. But the available corrections can't
simultaneously address those biases together (possibly they assume
addressing one reduces the negative impact of the other).

Also, I would imagine that the quantity and quality of the information
needed for these corrections could considerably impact the validity of
such corrections. In terms of quantity, for instance, if a good number
of the individual studies don't report their reliability estimates,
then depending on the situation, one may end up imputing them or
finding the reliability estimates from sources other than the primary
studies, if available. These could potentially create either
difficulty in implementing or uncertainty in terms of how well the
corrections might work.

In terms of quality, the corrections always work with (use as input)
*estimates* of, for instance, reliability or correlation reported in
the primary studies. As such, one may wonder how well those estimates
have done at representing their own *true values* in the first place
before being used to correct some source of bias. This again could
potentially create uncertainty in terms of how well the corrections
might work in practice.

Kind regards,
Reza


On Thu, May 25, 2023 at 9:01 PM Yefeng Yang via R-sig-meta-analysis
<r-sig-meta-analysis using r-project.org> wrote:
>
> Dear Tori
>
> The method mentioned is called bare-bones meta-analysis, which is a common method in the psychological meta-analysis (it is also called psychometric meta-analysis or called Hunter and Schmidt to remind the contributors of this method ).  The core idea is you need to correct for various artifacts in the effect size estimates (mostly in correlation but can be extended to other effect size measures). This includes, for example, using the reliability coefficient to account for measurement error (as suggested by your reviewer), and range restrictions. Schmidt and Hunter’s (2015) is the best reference for this. metafor can implement this with some tricks., But as far as I know there is a dedicated R package to do this, although I have not used it before - psychmeta package.
>
> Best,
> Yefeng
>
>
> ________________________________
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> on behalf of Tori Peña via R-sig-meta-analysis <r-sig-meta-analysis using r-project.org>
> Sent: Friday, 26 May 2023 1:51
> To: r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
> Cc: Tori Peña <Tori.Pena using stonybrook.edu>
> Subject: [R-meta] Question Re: Correcting for Statistical Artifacts
>
> Hello Listserv -
>
> I'm sending this question out again in case it slipped through the cracks:
>
> I am working on a multilevel meta-analysis and a reviewer asked us to
> correct for statistical artifacts (e.g., measurement error).  The reviewer
> cited Wiernik & Dahlke (2020) which includes some code using the metafor
> package.
>
> I am still unclear on a couple of things related to these analyses.  First,
> can one apply these analyses to nested models?  Second, one needs observed
> selection-effect *u* ratios to conduct these models but I am not sure how
> to get these values to consequently conduct the models.  I copied and
> pasted example code from Wiernik & Dahlke (2020) below for reference.  The
> "ux" and "uy" values are the selection-effect *u *ratios.
>
> ma_results_r <-
>   ma_r(ma_method = "ad",
>   rxyi = rxyi,
>   n = n,
>   rxx = rxxi,
>   ryy = ryyi,
>   ux = ux,
>   uy = uy,
>   correct_rxx = TRUE,
>   correct_ryy = TRUE,
>   correct_rr_x = TRUE,
>   correct_rr_y = TRUE,
>   indirect_rr_x = TRUE,
>   indirect_rr_y = TRUE,
>   data = data_r_bvirr)
>
> Any guidance would be greatly appreciated!  Thank you in advance for your
> time and energy on addressing this question.  I especially wanted to get
> input from this group since these analyses are new to me.
>
> Best,
> Tori
>
> --
> *Tori Peña, Ph.D. *(she/her/ella)
> Cognitive Psychology
> Dept. of Psychology
> Stony Brook University
> Stony Brook, NY 11790-2500
> [image: Stony Brook University logo]
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis