[R-meta] Publication bias with multivariate meta analysis

[Norman DAURELLE]

Dear Wolfgang, dear list members, 

I just finished reading the paper you advised me to read, and I have multiple questions. 

Page 9 paragraph 3 : “Overall, the Type-I error calibration of the funnel plot asymmetry tests degrades as the number of studies increases, the primary study sample size distribution decreases, and heterogeneity increases.” 

1°) Do these considerations apply when working with a slope? 
The article was based on Standardized Mean Difference, but some remarks should still apply to a funnel plot of a slope on the x-axis and standard error on the y-axis. The authors state in the discussion that they have not performed their analyses with other metrics than SMD and that more complex experimental designs than two-groups designs have not been investigated either, but they believe there should be similarities with other metrics. 

2°) would my meta-analysis have had a lower chance of producing a funnel plot displaying asymmetry if I had found 10 studies with datasets the same size as the dataset that was collected in the lab where I was an intern (1350 yield observations, vs. 640 for the second biggest sample size and under 45 for all other studies included)? 

3°) what can I do about it? 
My aim is to account truthfully for the diversity of the results in the scientific literature, while trying to show that since the species involved in the yield-disease slope I am dealing with are always the same (1 plant – 1 pathogen fungus), there necessarily is a “part of that slope” that is common to all the slope values that have been established. 

In practical terms: 

Should I try to variance-stabilize transform my data and perform the tests they perform? 
I am not sure I would know how to do that. 

Should I try to use a modified measure of precision (but the authors seem to recommend using variance-stabilizing transformation) and perform those tests ? 
Not sure I would know how to do that either. 

Is it “that big an issue” if my funnel plot is asymmetrical? 

Could I add a factor such as “country” to my dataset containing the slopes and use it as a moderator? 
I ask this because you wrote “Unaccounted for moderators that are related to the size of the estimates *and their SEs* could also create asymmetry.”, but if I understand it correctly, in my case, that would amount to having another predictor of yield when I get or create the yield-disease relationship slope. I sometimes did not have access to the original yield – disease severity measurement data. Therefore, if I understand it correctly, even if I added a moderator in my meta-analysis, it would not change the SEs of the slope estimates. 

I am going to check references 61 to 64 of Pustejovsky & Rodgers ‘s paper, and I will try and see what I can take from these papers. 

Once again, thank you for this reference. 

Best wishes, 
Norman 




De: "Norman DAURELLE" <norman.daurelle using agroparistech.fr> 
À: "Wolfgang Viechtbauer, SP" <wolfgang.viechtbauer using maastrichtuniversity.nl> 
Cc: "r-sig-meta-analysis" <r-sig-meta-analysis using r-project.org>, "Huang Wu" <huang.wu using wmich.edu> 
Envoyé: Mardi 31 Août 2021 00:11:28 
Objet: Re: [R-meta] Publication bias with multivariate meta analysis 

Dear Wolfgang, 

thank you for your explanation. I have not yet read the reference you advise me to read, but I will, and will ask further questions that may come to me after I have. 

Best wishes, 
Norman 


De: "Wolfgang Viechtbauer, SP" <wolfgang.viechtbauer using maastrichtuniversity.nl> 
�: "Norman DAURELLE" <norman.daurelle using agroparistech.fr> 
Cc: "Huang Wu" <huang.wu using wmich.edu>, "r-sig-meta-analysis" <r-sig-meta-analysis using r-project.org> 
Envoy�: Lundi 30 Ao�t 2021 11:43:53 
Objet: RE: [R-meta] Publication bias with multivariate meta analysis 

That's one of the main uses of a funnel plot: To see if it looks (roughly) symmetric or not. If it is not symmetric, then in essence this implies that there is a relationship between the estimates and whatever you put on the y-axis of the plot (typically the SEs). 

There are many possible reasons for why such a relationship could exist (i.e., why the plot is not symmetric). Publication bias is one of them. For certain effect size measures, there is also an inherent relationship between the observed effects and their sampling variances (and hence SEs), so in this case the funnel plot (and methods based on it) that put the SE (or some function thereof on the y-axis) are inherently flawed. To give one example: 

Pustejovsky, J. E., & Rodgers, M. A. (2019). Testing for funnel plot asymmetry of standardized mean differences. Research Synthesis Methods, 10(1), 57-71. https://doi.org/10.1002/jrsm.1332 

Unaccounted for moderators that are related to the size of the estimates *and their SEs* could also create asymmetry. 

In principle, one could also reason that asymmetry is to be expected if study authors are doing sample size calculations / power analyses before conducting their studies. 

I just saw that Gerta also replied and provided a useful reference. 

Best, 
Wolfgang 

>-----Original Message----- 
>From: Norman DAURELLE [mailto:norman.daurelle using agroparistech.fr] 
>Sent: Monday, 30 August, 2021 10:23 
>To: Viechtbauer, Wolfgang (SP) 
>Cc: Huang Wu; r-sig-meta-analysis 
>Subject: Re: [R-meta] Publication bias with multivariate meta analysis 
> 
>Dear list members, dear Huang and Wolfgang, 
> 
>thank you for explaining that there is no method for testing for publication bias, 
>or more accurately, for explaining that a relationship between observed effects 
>and their standard errors does not necessarily indicate publication bias (meaning 
>that there are other reasons why one could encounter such a relationship). 
> 
>Outside of Huang's question : does funnel plot asymetry necessarily indicate a 
>relationship between observed effects and their standard error ? 
> 
>I am going to have a deeper read at https://www.metafor-project.org/ but I would 
>be grateful for an answer. 
> 
>Best wishes, 
>Norman 

[[alternative HTML version deleted]] 


_______________________________________________ 
R-sig-meta-analysis mailing list 
R-sig-meta-analysis using r-project.org 
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis 

	[[alternative HTML version deleted]]