[R-meta] Sample-size weighting of estimates of response ratios?

[Viechtbauer, Wolfgang (NP)]

Glad to hear you figured it out.

As for "there is no solution to these problems": One could consider using the generalized Q-statistic estimator (DerSimonian & Kacker, 2007; Jackson et al., 2014) also with sample size weights. 

https://wviechtb.github.io/metafor/reference/rma.uni.html#specifying-the-model

And Eq. 7 in Bakbergenuly et al. (2020) is the algebraic version of the matrix equation for Var[b] I gave below for the simple case that X is just a column vector of 1s (i.e., for an 'intercept-only model').

Best,
Wolfgang

>-----Original Message-----
>From: Will Hopkins [mailto:willthekiwi using gmail.com]
>Sent: Sunday, 23 April, 2023 6:04
>To: 'R Special Interest Group for Meta-Analysis'
>Cc: Viechtbauer, Wolfgang (NP)
>Subject: RE: [R-meta] Sample-size weighting of estimates of response ratios?
>
>There's no need for anyone to respond to my previous message below. I
>mistakenly thought that somehow I could get proc mixed in SAS to estimate
>the mean effect and the heterogeneity from a single mixed model, in which
>the weighting of each study estimate was the sample size. But on a less
>careless reading of Bakbergenuly et al. (2020), I see that they used
>separate estimation equations for each of the mean effect, its confidence
>interval, the heterogeneity, and its confidence interval.  I have reproduced
>these equations with SAS one way and another. I used the usual inverse
>variance weighting in a mixed model to get the heterogeneity and its
>confidence interval; depending on sample sizes and so on, it's biased low,
>and the coverage is below optimal, but there is no solution to these
>problems. The mean effect is simply the sample-size weighted estimate, and
>its confidence interval comes from Equation 7 in Bakbergenuly et al.,
>combining the heterogeneity with sample sizes and study variances; these
>both perform better than the usual estimates.
>
>Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses of
>response ratios. BMC Med Res Methodol. 2020;20(1):263.
>doi:10.1186/s12874-020-01137-1
>
>Will
>
>-----Original Message-----
>From: Will Hopkins <willthekiwi using gmail.com>
>Sent: Friday, April 21, 2023 12:43 PM
>To: 'Viechtbauer, Wolfgang (NP)'
><wolfgang.viechtbauer using maastrichtuniversity.nl>
>Cc: 'R Special Interest Group for Meta-Analysis'
><r-sig-meta-analysis using r-project.org>
>Subject: RE: [R-meta] Sample-size weighting of estimates of response ratios?
>
>Thank you again, Wolfgang. I understand the use of the variance-covariance
>matrix to specify random effects, and although I learned matrix algebra more
>than 50 years ago, unfortunately I have not bothered with understanding the
>matrix algebra expressions for the solutions for the parameters in mixed
>models. I've treated them like a black box, but making sure I understand the
>sources of variation and their relationships (if any) with each other, and
>checking that I've got it right by doing simulations.
>
>So from what you have written below, I cannot understand whether or how the
>sampling variance of each study estimate is taken into account, when
>Bakbergenuly et al. say they used sample size as the weighting factor,
>rather than the inverse of the sampling variance, in their meta-analyses of
>the log of factor effects on means (response ratios or ratios of means). I
>would be really grateful if you can enlighten me in this respect, so I can
>work out how to do their style of meta-analysis with proc mixed in SAS to
>check myself with simulations that their approach is better, or rather, to
>work out how big the effect and the various standard deviations
>(between-subject in each study, measurement error in each study,
>heterogeneity between studies) need to be, and how small the sample size in
>each study needs to be, before their approach is better than using the
>inverse of the sampling variance.
>
>Will
>
>-----Original Message-----
>From: Viechtbauer, Wolfgang (NP)
><wolfgang.viechtbauer using maastrichtuniversity.nl>
>Sent: Friday, April 21, 2023 3:45 AM
>To: R Special Interest Group for Meta-Analysis
><r-sig-meta-analysis using r-project.org>
>Cc: Will Hopkins <willthekiwi using gmail.com>
>Subject: RE: [R-meta] Sample-size weighting of estimates of response ratios?
>
>One doesn't need to specify a covariance matrix (unless we are talking about
>situations where there are dependent estimates, but I don't think this is
>what we are discussing here). If one wants to use custom weights with the
>rma.uni() function, one just specifies them via the 'weights' argument. If
>one doesn't specify those weights, then the standard 1 / (vi + tau^2)
>weights are used (in a random/mixed-effects model). In either case, let W be
>diagonal with the weights along the diagonal, let X be the model matrix, and
>y the column vector with the effect size estimates. Then the model
>coefficients are estimated using weighted least squares (which is the same
>as maximum likelihood estimation in this case) with
>
>b = (X'WX)^-1 X'W y
>
>And the var-cov matrix of the coefficients is therefore given by
>
>Var[b] = (X'WX)^-1 X'W Var[y] WX (X'WX)^-1
>
>When using the default weights, Var[y] = W^-1, in which case the above
>simplifies to
>
>Var[b] = (X'WX)^-1
>
>When using custom weights, this is no longer true and one has to use the
>full equation above, where Var[y] is diagonal with vi + tau^2 along the
>diagonal.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis
>>[mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Will
>>Hopkins via R-sig-meta-analysis
>>Sent: Wednesday, 19 April, 2023 1:55
>>To: 'R Special Interest Group for Meta-Analysis'
>>Cc: Will Hopkins
>>Subject: Re: [R-meta] Sample-size weighting of estimates of response
>ratios?
>>
>>Thanks for this reply, Wolfgang. I've searched the metafor
>>documentation for "rma.uni" and for "weight matrix", but I can't see
>>how one specifies the covariance matrix when weighting is done only by
>>sample size. With the mixed model in SAS, weighting by the inverse of
>>the variances is done the long way by holding the residual for each
>>study estimate to its variance. (Doing it the elegant way is achieved
>>with a weight statement specifying the inverse of the variance, and
>>there is a single residual set to unity.) Presumably the standard error
>>of each estimate is somehow still included in the meta (how else could
>>you get meaningful uncertainty in the mean effect and an estimate of
>>heterogeneity?), even though the weighting is only by sample size, but
>>I can't see how. Do you still set the covariance matrix to a diagonal
>>of
>the variances, but you now include a weighting by sample size?
>>
>>Will
>>
>>-----Original Message-----
>>From: Viechtbauer, Wolfgang (NP)
>><wolfgang.viechtbauer using maastrichtuniversity.nl>
>>Sent: Sunday, April 16, 2023 11:11 PM
>>To: R Special Interest Group for Meta-Analysis
>><r-sig-meta-analysis using r-project.org>
>>Cc: Will Hopkins <willthekiwi using gmail.com>
>>Subject: RE: [R-meta] Sample-size weighting of estimates of response
>ratios?
>>
>>Dear Will,
>>
>>metafor allows the user to adjust the weights to any weights deemed
>>reasonable. See the 'weights' argument in rma.uni() and the 'W'
>>argument in
>>rma.mv() (in the latter case, one can specify an entire weight matrix).
>>
>>Best,
>>Wolfgang
>>
>>>-----Original Message-----
>>>From: R-sig-meta-analysis
>>>[mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Will
>>>Hopkins via R-sig-meta-analysis
>>>Sent: Sunday, 16 April, 2023 3:36
>>>To: 'R Special Interest Group for Meta-Analysis'
>>>Cc: Will Hopkins
>>>Subject: [R-meta] Sample-size weighting of estimates of response ratios?
>>>
>>>I know that metafor allows meta-analysis of response ratios (aka
>>>factor effects or ratios of means), but I can't find in the metafor
>>>documentation whether it's possible to weight the individual study
>>>estimates with their effective sample size rather than the usual
>>>inverse of the square of the standard error.  Bakbergenuly et al.
>>>(2020) recommended this approach to reduce the downward bias in the
>>meta-analyzed mean ratio and heterogeneity.
>>>I am not a user of metafor, but I need to be able to state whether
>>>it's available for a manuscript I am revising with a colleague (Dave
>>>Rowlands) about better approaches than standardization when
>>>meta-analyzing
>>means.
>>>Wolfgang, is it already available, or if not, do you intend to
>>>implement
>>it?
>>>
>>>We use SAS's proc mixed for metas, with the elegant method of setting
>>>the residual variance to unity, but I don't know how to adapt this
>>>method to weighting by sample size. If anyone on this list can
>>>enlighten me, that would also be cool, thank you.
>>>
>>>Will
>>>
>>>Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses
>>>of response ratios. BMC Med Res Methodol. 2020;20(1):263.
>>>doi:10.1186/s12874-020-01137-1