[R-meta] Predictive interval in MA with less than 10 studies

[Viechtbauer, Wolfgang (SP)]

And:

Wang, C. C., & Lee, W. C. (2019). A simple method to estimate prediction intervals and predictive distributions: Summarizing meta-analyses beyond means and confidence intervals. Research Synthesis Methods, 10(2), 255-266. https://doi.org/10.1002/jrsm.1345

But to add to this:

The issue of k and normality are a bit conflated here. If the distribution of true effects is non-normal, then k could be a million and a PI calculated under the assumption of normality is still garbage.

But if the distribution is normal (or approximately so), then k is relevant for getting an accurate estimate of tau^2 (which is what mostly determines the width of the PI, besides the SE of mu-hat).

As for the method of estimation: The same concerns apply whether one uses the method of moments, ML/REML, or Bayesian methods. Not sure why you think those concerns do not apply for the latter two types.

In general: I would consider all commonly-used methods for calculating a PI (including Bayesian methods) as rough approximations, regardless of k (well, I might have a lower bound on k, but that more generally applies to the use of RE models). They don't have nominal coverage properties, but are still useful to translate the estimate of tau^2 (which is difficult to interpret) into a range of 'plausible' effects one might see across many studies (including future ones).

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Tobias Saueressig
>Sent: Monday, 27 September, 2021 10:44
>To: Philippe Tadger
>Cc: r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Predictive interval in MA with less than 10 studies
>
>Dear Philippe,
>
>this might be of interest for
>you: https://journals.sagepub.com/doi/10.1177/0962280218773520
>
>Regards,
>
>Tobias
>
>Gesendet: Montag, 27. September 2021 um 10:34 Uhr
>Von: "Philippe Tadger" <philippetadger using gmail.com>
>An: "r-sig-meta-analysis using r-project.org" <r-sig-meta-analysis using r-project.org>
>Betreff: [R-meta] Predictive interval in MA with less than 10 studies
>Dear R-sig-MA community
>
>According to Cochrane manual: it's recommended to not trust in the PI
>when there are fewer than 10 studies because such calculation relies on
>the assumption of normality. Is there a way to check formally on each
>case when using less than 10 studies is not safe for PI calculation?. I
>can understand this limitation when the PI is calculated through a
>method that uses the methods of moments (or exact calculations like
>Riley 2001), but when the PI comes from a model that uses ML/REML (or
>iterative methods with identifiable likelihood) or Bayesian, such
>concern cannot exist. I would like to find confirmation or refutation of
>this idea.
>
>In advance,  your time and shared wisdom are appreciated.
>--
>Kind regards/Saludos cordiales
>*Philippe Tadger*
>ORCID <https://orcid.org/0000-0002-1453-4105>, Reseach Gate
><https://www.researchgate.net/profile/Philippe-Tadger>