[R-meta] Response Ratios in nested studies

[Reza Norouzian]

Hi Fred,

No, this is not possible. On the other hand, the basic summary
statistics to estimate both LRR and SMD are the same. So, you can
repeat your analysis with SMD, if you want.

Given that you mentioned you have overlapping info. for participants
(due to repeated measurement over time and outcome) in some studies,
it would be useful to construct some form of a within-study (or within
whatever level that is appropriate in your case) var-covariance matrix
for effect estimates in such studies and feed it into the rma.mv()
function.

Kind regards,
Reza

On Tue, Oct 19, 2021 at 12:08 PM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
>
> One final thought, is log response ratio (LRR) and SMD algebraically
> related? I mean can I convert my final result from LRR to SMD metric
> in case there is a conversion formula?
>
> Thanks,
> Fred
>
> On Tue, Oct 19, 2021 at 11:27 AM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
> >
> > I see, in my case these dynamics can in some studies happen together
> > (assignment of classrooms to treatments, treatments measured multiple
> > times, and at times, on additional outcomes). I think eventually the
> > guesstimate of ICC should alleviate the autocorrelation to some
> > extent.
> >
> > Thanks again,
> > Fred
> >
> > On Tue, Oct 19, 2021 at 11:18 AM James Pustejovsky <jepusto using gmail.com> wrote:
> > >
> > > I am not entirely sure. Tentatively, I would think that if the clustering comes from repeated measurements, then one would expect the ICC to be different than the ICC from a study where the clustering comes from having measured multiple individuals that are nested within higher-level units.
> > >
> > > On Tue, Oct 19, 2021 at 10:59 AM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
> > >>
> > >> Thanks! Regarding the guesstimate of ICC, does it matter, in terms of
> > >> ICC size, if the nested study is longitudinal or if it has measured
> > >> multiple outcomes (or both)?
> > >>
> > >> On Tue, Oct 19, 2021 at 10:52 AM James Pustejovsky <jepusto using gmail.com> wrote:
> > >> >
> > >> > Yes, that's right.
> > >> >
> > >> > On Tue, Oct 19, 2021 at 10:50 AM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
> > >> >>
> > >> >> Oh, I may have misunderstood the correct application of DEF. So, this
> > >> >> is a study-specific formula, right? That is, "n-lower" is the average
> > >> >> cluster size in each study not across all studies?
> > >> >>
> > >> >> DEF = (n-lower - 1) * ICC + 1
> > >> >>
> > >> >> Fred
> > >> >>
> > >> >> On Tue, Oct 19, 2021 at 10:35 AM James Pustejovsky <jepusto using gmail.com> wrote:
> > >> >> >
> > >> >> > Hi Fred,
> > >> >> >
> > >> >> > Do you have information about cluster sizes among the studies with nested structure? If you do, and if their average cluster sizes differ, then it would be better to use this information to calculate a unique DEF for each study. Even if you assume a common ICC, the DEF will not necessarily be identical for every study.
> > >> >> >
> > >> >> > James
> > >> >> >
> > >> >> > On Tue, Oct 19, 2021 at 10:28 AM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
> > >> >> >>
> > >> >> >> Absolutely, so I will proceed with the vi given by metafor::escalc()
> > >> >> >> and then multiply the vi given by escalc() by a common DEF with a
> > >> >> >> common ICC across the studies that have nested structure and then
> > >> >> >> change that common ICC and inspect the change in coefficients (I'll
> > >> >> >> probably do this on a null model without moderators).
> > >> >> >>
> > >> >> >> Once again, many thanks,
> > >> >> >> Fred
> > >> >> >>
> > >> >> >> On Tue, Oct 19, 2021 at 10:15 AM James Pustejovsky <jepusto using gmail.com> wrote:
> > >> >> >> >
> > >> >> >> > Hi Fred,
> > >> >> >> >
> > >> >> >> > Yes, it is definitely possible and sensible to combine the DEF correction with RVE meta-analysis. However, I think it may be important to use the initial DEF correction (and accompanying sensitivity analysis), even if it is only based on ballpark assumptions. Without it, studies with clustered samples will get an inordinately large amount of weight in the meta-analysis, leading to imprecise estimates of average effects and inflated estimates of between-study heterogeneity.
> > >> >> >> >
> > >> >> >> > James
> > >> >> >> >
> > >> >> >> > On Tue, Oct 19, 2021 at 10:07 AM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
> > >> >> >> >>
> > >> >> >> >> Dear Reza and James,
> > >> >> >> >>
> > >> >> >> >> Thank you so much for your, as always, valuable advice. Can we
> > >> >> >> >> possibly combine your two suggestions?
> > >> >> >> >>
> > >> >> >> >> I mean can we both correct the initial, incorrect sampling variances
> > >> >> >> >> and then apply the clubSandwich package?
> > >> >> >> >>
> > >> >> >> >> The reason is that finding the correct ICC is one issue, but then
> > >> >> >> >> assuming that ICC is going to be the same across the groups is another
> > >> >> >> >> issue which together make such a correction possibly a bit imprecise.
> > >> >> >> >>
> > >> >> >> >> Thanks much,
> > >> >> >> >> Fred
> > >> >> >> >>
> > >> >> >> >>
> > >> >> >> >> On Tue, Oct 19, 2021 at 9:30 AM James Pustejovsky <jepusto using gmail.com> wrote:
> > >> >> >> >> >
> > >> >> >> >> > Hi Fred,
> > >> >> >> >> >
> > >> >> >> >> > This is a good question. I am in the same boat as Reza, as I don't know of any methods work that examines the issue (though it seems like the sort of thing that must be out there?). I'm going to respond under the assumption that you don't have access to raw data and are just working with reported summary statistics from a set of studies, some or all of which ignored the clustering issue.
> > >> >> >> >> >
> > >> >> >> >> > My first thought would be to use the same sort of cluster-correction that is used for raw or standardized mean differences. The variance of the LRR is based on a delta method approximation, and it can be expressed as
> > >> >> >> >> >
> > >> >> >> >> > vi = se1^2 / m1^2 + se2^2 / m2^2,
> > >> >> >> >> >
> > >> >> >> >> > where se1 = sd1 / sqrt(n1) and se2 = sd2 / sqrt(n2) are the standard errors of the means in each group (calculated ignoring clustering, assuming a sample of independent observations). The issue with clustered data is that the usual standard errors are too small because of dependent observations. The usual way to correct the issue is to inflate the standard errors by the square root of the design effect, defined as
> > >> >> >> >> >
> > >> >> >> >> > DEF = (n-lower - 1) * ICC + 1,
> > >> >> >> >> >
> > >> >> >> >> > where n-lower is the number of lower-level observations per cluster (or the average number of observations per cluster, if there is variation in cluster size) and ICC is an intra-class correlation describing the proportion of the total variation in the outcome that is between clusters.
> > >> >> >> >> >
> > >> >> >> >> > If we assume that the ICC is the same in each group, then the design effect hits both standard errors the same way, and so we can just use
> > >> >> >> >> >
> > >> >> >> >> > vi = DEF * (se1^2 / m1^2 + se2^2 / m2^2),
> > >> >> >> >> >
> > >> >> >> >> > In some areas of application, it can be hard to find empirical information about ICCs, in which case you may just have to make some rough assumptions in calculating the DEF then conduct sensitivity analysis for varying values of ICC.
> > >> >> >> >> >
> > >> >> >> >> > If my initial assumption is wrong and you do have access to raw data, then the following recent article might be of help:
> > >> >> >> >> > https://doi.org/10.1002/sim.9226
> > >> >> >> >> >
> > >> >> >> >> > Best,
> > >> >> >> >> > James
> > >> >> >> >> >
> > >> >> >> >> > On Fri, Oct 15, 2021 at 9:00 PM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
> > >> >> >> >> >>
> > >> >> >> >> >> Hello All,
> > >> >> >> >> >>
> > >> >> >> >> >> I recently came across a post
> > >> >> >> >> >> (https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003330.html)
> > >> >> >> >> >> that discussed an issue that is relevant to my meta-analysis.
> > >> >> >> >> >>
> > >> >> >> >> >> In short, if some studies have nested structures, and the effect size
> > >> >> >> >> >> of interest is log response ratio (LRR), is there a way to adjust the
> > >> >> >> >> >> sampling variances (below) before modeling the effect sizes?
> > >> >> >> >> >>
> > >> >> >> >> >> vi = sd1i^2/(n1i*m1i^2) + sd2i^2/(n2i*m2i^2)
> > >> >> >> >> >>
> > >> >> >> >> >> Thank you,
> > >> >> >> >> >> Fred
> > >> >> >> >> >>
> > >> >> >> >> >> _______________________________________________
> > >> >> >> >> >> R-sig-meta-analysis mailing list
> > >> >> >> >> >> R-sig-meta-analysis using r-project.org
> > >> >> >> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis