[R-meta] Sample-size weighting of estimates of response ratios?

[Will Hopkins]

There's no need for anyone to respond to my previous message below. I
mistakenly thought that somehow I could get proc mixed in SAS to estimate
the mean effect and the heterogeneity from a single mixed model, in which
the weighting of each study estimate was the sample size. But on a less
careless reading of Bakbergenuly et al. (2020), I see that they used
separate estimation equations for each of the mean effect, its confidence
interval, the heterogeneity, and its confidence interval.  I have reproduced
these equations with SAS one way and another. I used the usual inverse
variance weighting in a mixed model to get the heterogeneity and its
confidence interval; depending on sample sizes and so on, it's biased low,
and the coverage is below optimal, but there is no solution to these
problems. The mean effect is simply the sample-size weighted estimate, and
its confidence interval comes from Equation 7 in Bakbergenuly et al.,
combining the heterogeneity with sample sizes and study variances; these
both perform better than the usual estimates.

Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses of
response ratios. BMC Med Res Methodol. 2020;20(1):263.
doi:10.1186/s12874-020-01137-1

Will

-----Original Message-----
From: Will Hopkins <willthekiwi using gmail.com> 
Sent: Friday, April 21, 2023 12:43 PM
To: 'Viechtbauer, Wolfgang (NP)'
<wolfgang.viechtbauer using maastrichtuniversity.nl>
Cc: 'R Special Interest Group for Meta-Analysis'
<r-sig-meta-analysis using r-project.org>
Subject: RE: [R-meta] Sample-size weighting of estimates of response ratios?

Thank you again, Wolfgang. I understand the use of the variance-covariance
matrix to specify random effects, and although I learned matrix algebra more
than 50 years ago, unfortunately I have not bothered with understanding the
matrix algebra expressions for the solutions for the parameters in mixed
models. I've treated them like a black box, but making sure I understand the
sources of variation and their relationships (if any) with each other, and
checking that I've got it right by doing simulations.  

So from what you have written below, I cannot understand whether or how the
sampling variance of each study estimate is taken into account, when
Bakbergenuly et al. say they used sample size as the weighting factor,
rather than the inverse of the sampling variance, in their meta-analyses of
the log of factor effects on means (response ratios or ratios of means). I
would be really grateful if you can enlighten me in this respect, so I can
work out how to do their style of meta-analysis with proc mixed in SAS to
check myself with simulations that their approach is better, or rather, to
work out how big the effect and the various standard deviations
(between-subject in each study, measurement error in each study,
heterogeneity between studies) need to be, and how small the sample size in
each study needs to be, before their approach is better than using the
inverse of the sampling variance. 

Will

-----Original Message-----
From: Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer using maastrichtuniversity.nl>
Sent: Friday, April 21, 2023 3:45 AM
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis using r-project.org>
Cc: Will Hopkins <willthekiwi using gmail.com>
Subject: RE: [R-meta] Sample-size weighting of estimates of response ratios?

One doesn't need to specify a covariance matrix (unless we are talking about
situations where there are dependent estimates, but I don't think this is
what we are discussing here). If one wants to use custom weights with the
rma.uni() function, one just specifies them via the 'weights' argument. If
one doesn't specify those weights, then the standard 1 / (vi + tau^2)
weights are used (in a random/mixed-effects model). In either case, let W be
diagonal with the weights along the diagonal, let X be the model matrix, and
y the column vector with the effect size estimates. Then the model
coefficients are estimated using weighted least squares (which is the same
as maximum likelihood estimation in this case) with

b = (X'WX)^-1 X'W y

And the var-cov matrix of the coefficients is therefore given by

Var[b] = (X'WX)^-1 X'W Var[y] WX (X'WX)^-1

When using the default weights, Var[y] = W^-1, in which case the above
simplifies to

Var[b] = (X'WX)^-1

When using custom weights, this is no longer true and one has to use the
full equation above, where Var[y] is diagonal with vi + tau^2 along the
diagonal.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis
>[mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Will 
>Hopkins via R-sig-meta-analysis
>Sent: Wednesday, 19 April, 2023 1:55
>To: 'R Special Interest Group for Meta-Analysis'
>Cc: Will Hopkins
>Subject: Re: [R-meta] Sample-size weighting of estimates of response
ratios?
>
>Thanks for this reply, Wolfgang. I've searched the metafor 
>documentation for "rma.uni" and for "weight matrix", but I can't see 
>how one specifies the covariance matrix when weighting is done only by 
>sample size. With the mixed model in SAS, weighting by the inverse of 
>the variances is done the long way by holding the residual for each 
>study estimate to its variance. (Doing it the elegant way is achieved 
>with a weight statement specifying the inverse of the variance, and 
>there is a single residual set to unity.) Presumably the standard error 
>of each estimate is somehow still included in the meta (how else could 
>you get meaningful uncertainty in the mean effect and an estimate of 
>heterogeneity?), even though the weighting is only by sample size, but 
>I can't see how. Do you still set the covariance matrix to a diagonal 
>of
the variances, but you now include a weighting by sample size?
>
>Will
>
>-----Original Message-----
>From: Viechtbauer, Wolfgang (NP)
><wolfgang.viechtbauer using maastrichtuniversity.nl>
>Sent: Sunday, April 16, 2023 11:11 PM
>To: R Special Interest Group for Meta-Analysis 
><r-sig-meta-analysis using r-project.org>
>Cc: Will Hopkins <willthekiwi using gmail.com>
>Subject: RE: [R-meta] Sample-size weighting of estimates of response
ratios?
>
>Dear Will,
>
>metafor allows the user to adjust the weights to any weights deemed 
>reasonable. See the 'weights' argument in rma.uni() and the 'W'
>argument in
>rma.mv() (in the latter case, one can specify an entire weight matrix).
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis
>>[mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Will 
>>Hopkins via R-sig-meta-analysis
>>Sent: Sunday, 16 April, 2023 3:36
>>To: 'R Special Interest Group for Meta-Analysis'
>>Cc: Will Hopkins
>>Subject: [R-meta] Sample-size weighting of estimates of response ratios?
>>
>>I know that metafor allows meta-analysis of response ratios (aka 
>>factor effects or ratios of means), but I can't find in the metafor 
>>documentation whether it's possible to weight the individual study 
>>estimates with their effective sample size rather than the usual 
>>inverse of the square of the standard error.  Bakbergenuly et al.
>>(2020) recommended this approach to reduce the downward bias in the
>meta-analyzed mean ratio and heterogeneity.
>>I am not a user of metafor, but I need to be able to state whether 
>>it's available for a manuscript I am revising with a colleague (Dave
>>Rowlands) about better approaches than standardization when 
>>meta-analyzing
>means.
>>Wolfgang, is it already available, or if not, do you intend to 
>>implement
>it?
>>
>>We use SAS's proc mixed for metas, with the elegant method of setting 
>>the residual variance to unity, but I don't know how to adapt this 
>>method to weighting by sample size. If anyone on this list can 
>>enlighten me, that would also be cool, thank you.
>>
>>Will
>>
>>Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses 
>>of response ratios. BMC Med Res Methodol. 2020;20(1):263.
>>doi:10.1186/s12874-020-01137-1