[R-meta] Outlier and influential case analysis for multilevel meta-analysis with RVE

[James Pustejovsky]

Unfortunately I don't have any suggestions for readings about this issue.
Perhaps other on the listserv know of good readings?
James

On Mon, Aug 26, 2024 at 12:54 AM Maximilian Steininger <
maximilian.steininger using univie.ac.at> wrote:

> Dear James,
>
> Thanks a lot for your input. Then I will do the diagnostics before
> applying RVE!
>
> I agree that it’s quite challenging to define what an outlier is in the
> multilevel context. Since I have very few effect sizes per study (max. 3,
> mostly 1) and a big between-study (and not so much within-study)
> heterogeneity, my approach would be to identify outlying effect sizes with
> respect to overall distribution. Do you have any suggestions for further
> reading on this topic?
>
> Best and many thanks!
>
> Max
> ——
>
> Mag. Maximilian Steininger
>   PhD candidate
>
>   Social, Cognitive and Affective Neuroscience Unit
>   Faculty of Psychology
>   University of Vienna
>
>   Liebiggasse 5
>   1010 Vienna, Austria
>
>   e: maximilian.steininger using univie.ac.at
>   w: http://scan.psy.univie.ac.at
>
> > Am 24.08.2024 um 14:40 schrieb James Pustejovsky via R-sig-meta-analysis
> <r-sig-meta-analysis using r-project.org>:
> >
> > I think it makes sense to do the analysis of outliers and influential
> cases
> > before applying RVE. One way to think about this approach is that you are
> > examining the assumptions _of the working model_, to understand the
> extent
> > to which those assumptions are reasonable, even if you will later use RVE
> > to protect against model misspecification.
> >
> > I think this approach is advantageous because it gives access to a richer
> > set of diagnostic tools, whereas the other approach is just a single
> > rule-of-thumb (one which I don't think has a strong statistical rationale
> > in the first place).
> >
> > A further challenge here that I don't think has been addressed thoroughly
> > in the meta-analysis methods literature is how to think about outliers in
> > the multilevel context. When there is both between-study and within-study
> > variation, one could imagine there being outlying studies, outlying
> effect
> > sizes with respect to the overall distribution, or outlying effect sizes
> > relative to the distribution of effects within the same study. Perhaps
> > others on the list know of guidance about how to diagnose these features.
> >
> > Best,
> > James
> >
> > On Fri, Aug 23, 2024 at 6:09 AM Maximilian Steininger via
> > R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
> >
> >> Dear all,
> >>
> >> I am conducting a three-level meta-analysis where I have different
> >> dependency structures in my data. I model the dependency by
> approximating
> >> the var-cov matrix, followed by estimating a three-level model and then
> I
> >> apply robust variance estimation to compute my outcome (as suggested
> e.g.
> >> here:
> >>
> https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures
> >> <
> >>
> https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures
> >>> )
> >>
> >> I wanted to do some sensitivity analysis on the model by running outlier
> >> and influential diagnostics. However, most of the proposed diagnostics
> do
> >> not work on "robust.rma" objects.
> >>
> >> So far I did some model diagnostics by calculating cook's distance and
> hat
> >> values for my robust model (see e.g.,
> >> https://wviechtb.github.io/metafor/reference/influence.rma.mv.html <
> >> https://wviechtb.github.io/metafor/reference/influence.rma.mv.html>).
> But
> >> as far as I am concerned these "only" give me information on influential
> >> cases and not outliers.
> >>
> >> What is the best approach to check for outliers when using robust
> models?
> >> Are the two options below a sensible approach to check for outliers?
> >>
> >> According to this source a possible but rather conservative approach is
> to
> >> label all studies as outliers that have confidence intervals that do not
> >> overlap with the confidence interval of the pooled effect. (see:
> >>
> https://cjvanlissa.github.io/Doing-Meta-Analysis-in-R/detecting-outliers-influential-cases.html
> >> <
> >>
> https://cjvanlissa.github.io/Doing-Meta-Analysis-in-R/detecting-outliers-influential-cases.html
> >>> ).
> >> Is it a feasible option to perform outlier diagnostics for the
> non-robust
> >> model as suggested e.g. by Viechtbauer & Cheung (2010;
> 10.1002/jrsm.11). My
> >> approach here would be to identify outliers based on the non-robust
> model
> >> --> exclude the outliers --> rerun the whole analysis without the
> outliers
> >> (i.e., approximate var-cov matrix, estimate three-level model, apply
> robust
> >> variance estimation for the subset of studies).
> >> Or are there other, more elegant ways to do this?
> >>
> >> Best and many thanks!
> >> ——
> >>
> >> Mag. Maximilian Steininger
> >>  PhD candidate
> >>
> >>  Social, Cognitive and Affective Neuroscience Unit
> >>  Faculty of Psychology
> >>  University of Vienna
> >>
> >>  Liebiggasse 5
> >>  1010 Vienna, Austria
> >>
> >>  e: maximilian.steininger using univie.ac.at
> >>  w: http://scan.psy.univie.ac.at
> >>
> >>
> >>        [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> >> To manage your subscription to this mailing list, go to:
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >>
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> > To manage your subscription to this mailing list, go to:
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>

	[[alternative HTML version deleted]]