[R-meta] Fixed vs Random Effects

[Michael Dewey]

Comments in line. Perhaps even more necessary than usual to stress these 
are personla views and others will differ.

On 11/04/2018 23:59, Célia Sofia Moreira wrote:
> Dear all,
> 
> I'm needing your help on the decision between fixed- or random-effects. I
> know that most of you are reviewers in top and respectable journals on
> meta-analysis, and so I will take your opinion very seriously. The question
> is the following:
> 
> My "favourite" papers recommend the use of random effects when you want to
> make inferences about the average effect to the entire population of
> studies from which the included studies are assumed to be a random
> selection (including "studies that have been conducted, that could have
> been conducted, or that may be conducted in the future"). Others (Cochrane)
> recommend the use of random-effect when samples/experiments/designs/...
> have different features. All of them say that the choice should not be
> decided on the basis of presence/absence of heterogeneity, and the
> researcher should decide on the type of inference desired before examining
> the data.

That summary is what I believe too and I think the last sentence is one 
which is very important.

> 
> Papers included in 'my' meta-analysis have very different
> samples/experimental features, as the majority of studies in social
> sciences. Moreover, I consider that is advantageous to make inferences to
> the entire population, instead of making inferences only to the set of
> studies included in 'my' meta-analysis; it is a wider approach. Therefore,
> I decided to perform random-effects models. In most cases, the results
> showed only small heterogeneity (and thus the results for fixed effects are
> similar).
> 
> Now, a co-author disagrees with my point of view and says that the
> meta-analysis should be performed using fixed-effects models because (his
> main reasons):
> 1) "larger studies should have more weight" (sample sizes range from 25 to
> 65),
> 2) "choosing a random-effects model introduces an error in each study",
> 3) "fixed effects provide narrower CI intervals and, as such, more precise
> results".

Point 1 is indeed true. I do not understand point 2. Point 3 is also 
true but misses the point as to whether that narrowness is appropriate 
or not. If you want to choose a third option there is the method by 
Hemni and Copas which is available in  metafor. This was designed for 
situations of small study bias and basically gives you the fixed effect 
summary (which gets rid of point 1) while making the CI more wide.

> 
> He also gave me a reference of an article that was published in the same
> journal we are planning to submit 'our' meta-analysis, in which
> fixed-effects were preferred. The authors used the following argument:
> 
> "Studies on the effect of medications were combined using a fixed-effect
> model (Borenstein et al., 2010). We expected the final model to include
> only a small number of studies and estimation of random-effects models with
> few studies has been shown to be unreliable (Guolo and Varin, 2017).
> However, random-effects models were carried out in a sensitivity analysis."

It is true that the estimate of tau^2 is quite imprecise but I would 
have thought it more logical to do the analyses the other way round 
(random primary, fixed sensitivity).

There is also the issue f whether in the face of extreme heterogeneity 
it makes sense to give any summary estimate at all. I recently reviewed 
an article where they used random effects to combine two estimates, one 
from each sex. Apart from the issue of whether you can generalise to a 
population of other sexes there were some pairs of estimates which 
clearly looked different and where combining them obscured rather than 
illuminated.

> 
> I have confirmed that results from random- and fixed-effects models are
> similar in most of the cases (usually <= .01; narrower CI but the
> significance does not change), and even when the difference is higher
> (=.04) there is no "small-studies effect" (i.e., small studies are not
> consistently more positive, or negative).
> 
> What is your opinion on his arguments and on the argument used in that
> paper (i.e., the estimation of fixed-effects models is more reliable than
> the random-effects when there are only few studies)?
> 
> Kind regards,
> celia
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 

-- 
Michael
http://www.dewey.myzen.co.uk/home.html