[R-meta] Publication bias with multivariate meta analysis

[Dr. Gerta Rücker]

Dear Norman,

Please find my response inline below. (I skipped some lines of your e-mail).

Am 31.08.2021 um 00:07 schrieb Norman DAURELLE:
>
> Dear Gerta, dear list members
>
> thank you for your answer. I read the paper you indicated, and it was 
> really interesting for me to learn about what is discussed in it.
>
> In a way it gives me more questions than answers : I used to think 
> that heterogeneity was required for a meta-analysis. In my mind, if 
> there was no heterogeneity, there was no point in even using the 
> meta-analytic method (I thought no heterogeneity more or less meant 
> all studies "reported the same result" / that they all "agreed").
This is interesting. Yes, heterogeneity has to be expected, see Julian 
Higgins's paper https://academic.oup.com/ije/article/37/5/1158/871288 . 
However, it never came to my mind that heterogeneity was *required*. Of 
course, it is not: in cases where there is no heterogeneity at all 
(probably true only in a minority of research questions), meta-analysis 
makes much sense, as you can estimate the true effect with much more 
precision as in each single study.
>
> [...]
>
> So : after reading the paper you suggested I read, is it possible that 
> what I observed in the asymmetry of my funnel plot was actually mainly 
> the heterogeneity of the data (say, differences in slopes due to 
> temperature, or rainfall, or soil, etc... conditions in different 
> studies carried out in different countries) ?
>
> I still believe that there might be a slight publication bias (because 
> we expect yield to decrease with disease severity, and if the data 
> does not show that we will tend to think that we did not observe "the 
> effect of disease severity, all else being equal"), but I am not so 
> sure anymore that publication bias could be the main reason why the 
> funnel plot of this meta-analysis is asymmetrical.
>
> It seems that there may be more than one reason why it is, and some of 
> these reasons I might (from what I understand now, but I might still 
> be wrong) apparently never discover.
>
> However, I would definetly like to be able to offer possible reasons 
> why it is asymmetrical, and what this means.

I just saw Michael's example what might have happened in a different 
area, and in fact this seems to be a general pattern: Small studies tend 
to show greater effects. As I am working in the medical field, I know 
possible explanations from there (they cannot be well separated from 
each other):

- Small studies often have been the first studies for the given research 
question, conducted without funding, were less regularized, and had 
particularly selected observations;

- Small studies consider more homogeneous study populations (in health 
science therapy studies these are often subjects easier to treat).

This phenomenon has also be called the Proteus effect 
https://journals.plos.org/plosclinicaltrials/article?id=10.1371/journal.pctr.0010036 
.

As I don't know your field, it is difficult for me to guess what might 
be behind the funnel plot asymmetry in your case.

Best,

Gerta

>
> Sorry for this long and not very to-the-point e-mail,
>
> I would very much be grateful for an answer.
>
> Best wishes,
> Norman
>
>
>
>
>
> ------------------------------------------------------------------------
> *De: *"Dr. Gerta Rücker" <ruecker using imbi.uni-freiburg.de>
> *À: *"Norman DAURELLE" <norman.daurelle using agroparistech.fr>, "Wolfgang 
> Viechtbauer, SP" <wolfgang.viechtbauer using maastrichtuniversity.nl>
> *Cc: *"r-sig-meta-analysis" <r-sig-meta-analysis using r-project.org>, 
> "Huang Wu" <huang.wu using wmich.edu>
> *Envoyé: *Lundi 30 Août 2021 11:40:23
> *Objet: *Re: [R-meta] Publication bias with multivariate meta analysis
>
> Dear Norman,
>
> If there is funnel plot asymmtery, there is always some relation between
> observed effects and their standard errors, the question is what causes
> this relationship. Possible causes are discussed in Sterne et al.
> (2011), see https://www.bmj.com/content/343/bmj.d4002
>
> Best wishes,
>
> Gerta
>
> Am 30.08.2021 um 10:22 schrieb Norman DAURELLE:
> > Dear list members, dear Huang and Wolfgang,
> >
> > thank you for explaining that there is no method for testing for 
> publication bias, or more accurately, for explaining that a 
> relationship between observed effects and their standard errors does 
> not necessarily indicate publication bias (meaning that there are 
> other reasons why one could encounter such a relationship).
> >
> > Outside of Huang's question : does funnel plot asymetry necessarily 
> indicate a relationship between observed effects and their standard 
> error ?
> >
> > I am going to have a deeper read at [ 
> https://www.metafor-project.org/ | https://www.metafor-project.org/ ] 
> but I would be grateful for an answer.
> >
> > Best wishes,
> > Norman
> >
> >
> > De: "Wolfgang Viechtbauer, SP" 
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> > À: "Huang Wu" <huang.wu using wmich.edu>, "r-sig-meta-analysis" 
> <r-sig-meta-analysis using r-project.org>
> > Envoyé: Samedi 28 Août 2021 15:37:20
> > Objet: Re: [R-meta] Publication bias with multivariate meta analysis
> >
> > Dear Huang,
> >
> > Please find my comments below.
> >
> > Best,
> > Wolfgang
> >
> >> -----Original Message-----
> >> From: R-sig-meta-analysis 
> [mailto:r-sig-meta-analysis-bounces using r-project.org] On
> >> Behalf Of Huang Wu
> >> Sent: Saturday, 28 August, 2021 3:19
> >> To: r-sig-meta-analysis using r-project.org
> >> Subject: [R-meta] Publication bias with multivariate meta analysis
> >>
> >> Hi all,
> >>
> >> I am conducting a multivariate meta-analysis using rma.mv. I want 
> to test for
> >> publication bias.
> >> I noticed in a previous post, Dr. Pustejovsky provided the 
> following code for
> >> Egger’s test.
> >>
> >> egger_multi <- rma.mv(yi = yi, V = sei^2, random = ~ 1 | 
> studyID/effectID,
> >> mods = ~ sei, data = dat)
> >> coef_test(egger_multi, vcov = "CR2")
> >>
> >> Because I conducted a multivariate meta-analysis assuming rho = 
> 0.8, I wonder for
> >> the Egger’s test, Do I need to let V equals to the imputed 
> covariance matrix?
> >> Would anyone help me to see if my following code is correct? Thanks.
> >>
> >> V_listm <- impute_covariance_matrix(vi = meta$dv,
> >> cluster = meta$Study.ID,
> >> r = 0.8)
> >> egger_multi <- rma.mv(yi =Cohen.s.d, V = V_listm, random = ~ 1 | 
> Study.ID/IID,
> >> mods = ~ sqrt(dv), data = meta)
> >> coef_test(egger_multi, vcov = "CR2")
> > If you used such an approximate V matrix for your analyses, then I 
> would also use this in this model.
> >
> >> Also, I have tried V = V_listm and V = dv, but it gave me different 
> results. When
> >> I use V = V_Vlistm, my results suggest the effect was no longer 
> statistically
> >> significant but when I use V = dv, my result is still significant.
> >> Does that mean my results were sensitive to the value of rho? Thanks.
> > Yes, although it's not clear to me what exactly you mean by "the 
> effect". The coefficient corresponding to 'sqrt(dv)'?
> >
> >> By the way, does anyone have any suggestions/codes for other 
> methods of testing
> >> publication bias? Many thanks.
> > Just a pedantic note: There are no methods for testing for 
> publication bias. One can for example test if there is a relationship 
> between the observed effects and their standard errors (as done 
> above), which could result from publication bias, but there could be 
> other explanations for such a relationship besides publication bias.
> >
> > This aside, one can also examine if there is a relationship at the 
> study level (not at the level of the individual estimates, as done 
> above). A simple approach for this would be to aggregate the estimates 
> to the study level, using the aggregate() function. In fact, at that 
> point, you could apply all of the methods available in metafor or 
> other packages related to the issue of publication bias (including 
> things like trim-and-fill, selection models, and so on).
> >
> > Best,
> > Wolfgang
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis using r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis using r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
> -- 
>
> Dr. rer. nat. Gerta Rücker, Dipl.-Math.
>
> Institute of Medical Biometry and Statistics,
> Faculty of Medicine and Medical Center - University of Freiburg
>
> Zinkmattenstr. 6a, D-79108 Freiburg, Germany
>
> Mail:     ruecker using imbi.uni-freiburg.de
> Homepage: 
> https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker

-- 

Dr. rer. nat. Gerta Rücker, Dipl.-Math.

Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center - University of Freiburg

Zinkmattenstr. 6a, D-79108 Freiburg, Germany

Mail:     ruecker using imbi.uni-freiburg.de
Homepage: https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker


	[[alternative HTML version deleted]]