[R-meta] Reproducing results using regtest in metafor

[Viechtbauer, Wolfgang (SP)]

Chiming in very late here (when in fact the question has essentially been resolved), but just to point out that I wasn't ignoring you, Alex, I was just out of the office in August.

And indeed, regtest() refits the model completely, so if you first fit a 'standard' meta-analytic model and then use regtest(..., model="lm"), the initial model fit isn't relevant, except that it determines what other covariates are included in the model besides the one that is added by regtest().

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Sutton, Alex (Prof.)
Sent: Thursday, 15 August, 2019 15:43
To: James Pustejovsky
Cc: DOLEMAN, Brett (UNIVERSITY HOSPITALS OF DERBY AND BURTON NHS FOUNDATION TRUST); r-sig-meta-analysis using r-project.org; Freeman, Suzanne C. (Dr.); Sutton, Alex (Prof.)
Subject: Re: [R-meta] Reproducing results using regtest in metafor

Thanks James

Again, your response is super helpful.

(I did have a dig into the code of regtest but wasn't confident I knew enough about it to reliably unpick it). The point you make about coherence is important. It is interesting to note that the multiplicative error of the test as specified (originally by Egger) was due to it being conceived with respect to a Galbraith plot (which has standardised effect on one axis so regression is "divided through" by the standard error) not any real belief about the error structure - but it has stuck and does deal with the scenario of underdispersion (less variability than expected by chance even under fixed effect) which could occur if the most extreme study results under a fixed effect assumption have been suppressed.

So while the test as regtest conducts it does not mix errors, there is still the issue that the analysis strategy does (i.e. why use different error structures for baseline risk on its own and when adding in the standard error for asymmetry testing?). One thing in favour of the "2-stage" analysis is that it allows the construction of a funnel based on the residuals to see how its appearance changes as a result of adjusting for the initial covariate.

Thanks again for all your help - I now understand what is going on and thus you have comprehensively answered my question.

Very best wishes

Alex
________________________________
From: James Pustejovsky <jepusto using gmail.com>
Sent: 14 August 2019 19:50
To: Sutton, Alex (Prof.) <ajs22 using leicester.ac.uk>
Cc: Michael Dewey <lists using dewey.myzen.co.uk>; r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>; DOLEMAN, Brett (UNIVERSITY HOSPITALS OF DERBY AND BURTON NHS FOUNDATION TRUST) <brett.doleman using nhs.net>; Freeman, Suzanne C. (Dr.) <suzanne.freeman using leicester.ac.uk>
Subject: Re: [R-meta] Reproducing results using regtest in metafor

Alex,

Looking at the source code for regtest.rma, it looks like my previous syntax can be simplified even further, to simply:
lm_fit <- lm(yi ~ m.c + sqrt(vi), data = ma.dataset, weights = 1 / vi)
summary(lm_fit)

The random effects structure / weighting scheme used to fit the initial metaregression is simply disregarded when re-fitting the model. I think that's reasonable because it forces one to make a coherent assumption about the structure of the errors in the regression. In contrast, the other approach involves fitting the first model based on random effects weighting, and then fitting the Egger regression on the residuals, but using a different weighting scheme. I can't think of any data generating process where this would be an efficient way to fit the model or a valid approach for inference. Thus, it would seem preferable to fit the model all at once, using assumptions that can at least be clearly stated.

James

On Mon, Aug 12, 2019 at 7:34 AM Sutton, Alex (Prof.) <ajs22 using leicester.ac.uk<mailto:ajs22 using leicester.ac.uk>> wrote:
Dear James

Thank you for taking the time to think about this and write. I think your approach of writing the model out is a good one for discussing exactly what is going on "under the hood".

Your justification of the degrees of freedom seems sound - much appreciated.

I am still a little uncertain about the model specification because an additive random effect is specified in the initial meta-regression using the rma command. But then a multiplicative error is specified in the regtest command. These probably dont have any bearing on the degrees of freedom, but I would greatly appreciate it if someone could explain how these are jointly implemented.

With very best wishes

Alex

________________________________
From: James Pustejovsky <jepusto using gmail.com<mailto:jepusto using gmail.com>>
Sent: 09 August 2019 19:04
To: Sutton, Alex (Prof.) <ajs22 using leicester.ac.uk<mailto:ajs22 using leicester.ac.uk>>
Cc: Michael Dewey <lists using dewey.myzen.co.uk<mailto:lists using dewey.myzen.co.uk>>; r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org> <r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>>; DOLEMAN, Brett (UNIVERSITY HOSPITALS OF DERBY AND BURTON NHS FOUNDATION TRUST) <brett.doleman using nhs.net<mailto:brett.doleman using nhs.net>>; Freeman, Suzanne C. (Dr.) <suzanne.freeman using leicester.ac.uk<mailto:suzanne.freeman using leicester.ac.uk>>
Subject: Re: [R-meta] Reproducing results using regtest in metafor

Option 1 amounts to fitting the model

Y_i = b0 + b1 m.c_i + b2 se_i + e_i

with the assumption that Var(e_i) = sigma^2 v_i for unknown sigma^2.  It can be fit using the nlme package using the following syntax (building off of the previous code):

library(nlme)
gls_fit <- gls(yi ~ m.c + sqrt(vi), data = ma.dataset, weights = varFixed(~ vi))
summary(gls_fit)

I think the correct df would be 27 in this case.

With option 2, the residuals from the first stage have lost 2 degrees of freedom (down to 28), and then they lose two more with the second stage fit (because the intercept is re-estimated when it should be constrained to zero).

James

On Fri, Aug 9, 2019 at 8:24 AM Sutton, Alex (Prof.) <ajs22 using leicester.ac.uk<mailto:ajs22 using leicester.ac.uk>> wrote:
Dear Michael

Many thanks for taking the time to write. (And let me thank you for all your hard work on the R meta-analysis web page - this is a fantastic resource I use all the time)

I think your suggestion is very plausible and I was thinking along related lines. Then I convinced myself that the df used in first regression was reflected in the se of the residuals (in a very "hand wavy" sort of way!) .

As you say, the issue then focuses on which is "better" - my concern with option 1 is that I don't know how to reproduce it outside of the package - but as you say it may well be theoretically superior.

I am also hoping Wolfgang can supply some definitive insight!

Thanks again

Alex