[R-meta] mean-variance relationships introduces additional heterogeneity, how?

[James Pustejovsky]

Luke,

As a starting point, I would suggest creating graphs across _all_ of
the control groups, just treating every group as a data point and
ignoring the nesting structure. Sure, there might be some nuance to
the over-time relation within studies versus the between-study
relations. But as a first step, I think it would be reasonable to just
look at the mean-variance relationships across the whole dataset.

James

On Mon, Oct 25, 2021 at 12:02 PM Luke Martinez <martinezlukerm using gmail.com> wrote:
>
> Thanks for the clarification. Regarding the plots, you're saying that
> I plot the sds of the control groups against the means of the control
> groups **in each study** or **across the studies**? (Again, my studies
> are longitudinal.)
>
> If in each study, then for studies that provide at least three time
> points, I can plot the Means (m_itj) and SDs (s_itj) of proportions
> for each group i at time t in study j across the three (or more) time
> points to see if there is a positive relationship between m_itj and
> s_itj .
>
> If across the studies, plotting the mean-sd relation across
> longitudinal studies would seem to require that the same time points
> across the studies align in terms of their time intervals. In other
> words, the mean-sd relation for each group across longitudinal studies
> could be contaminated by the differences in time intervals for the
> testing occasions.
>
> This might be a reason to inspect this relation a bit more locally
> (perhap within studies). On the other hand, to do so, I'll need to
> have studies with at least three time points. Thus, if I have studies
> that only followed a pre-post-control design, then exploring that
> relation locally may not be possible.
>
> Aside from alignment of time points across the studies, there may be
> other differences across the studies in terms of treatment, or setting
> etc. that affect the mean-sd relationship which again makes it
> difficult to explore the mean-sd relationship across the studies.
>
> Perhaps, one could argue that because anything that affects the mean,
> affects the SD (in some way), then, such a relationship can still be
> explored across the studies.
>
> I would be curious to know your thoughts on these reflections?
>
> Luke
>
> On Mon, Oct 25, 2021 at 7:50 AM James Pustejovsky <jepusto using gmail.com> wrote:
> >
> > Hi Luke,
> >
> > Whoop, I switched notation halfway through writing my reply and evidently missed a couple of spots. Yes, for pi_Ai read m_Ai and for pi_Bi read m_Bi.
> >
> > As far as graphical diagnostics, it's a bit easier in your case than in the poisson-gamma model. Since you're working with proportions, I would suggest graphing the sds of the control groups against the means of the control groups (and the same for treatment groups). If you have some outcomes that are raw counts, you'd first want to translate the the summary statistics into proportions by using m_i / Ti and sd_i / Ti.
> >
> > Actually, if you know the number of trials going into each proportion, then one better than plotting sd_Ai versus m_Ai would be to factor out the number of trials, so plot sd_Ai * sqrt(Ti) versus m_Ai.
> >
> > Another potentially useful diagnostic is just to plot each effect size metric (SMD and response ratio) against m_Ai and against Ti. If my speculative model is in the ballpark, we would expect to see that the SMD would be moderated by both m_Ai and Ti.
> >
> > James
> >
> > Again since you're working
> >
> > James
> >
> >
> > On Sun, Oct 24, 2021 at 6:08 PM Luke Martinez <martinezlukerm using gmail.com> wrote:
> > >
> > > Many thanks James!
> > >
> > > Just to make sure, do pi_Ai and pi_Bi represent (in my participants
> > > taking a test context) the proportion of correct responses in each
> > > group (treatment vs. control) respectively?
> > >
> > > We're saying that even though these proportions (pi_Ai and pi_Bi) have
> > > a constant and linear relation via the multiplicative coefficient
> > > lambda in and across all the studies such that the treatment effect is
> > > the same across all the studies, we still get varying SMDs across
> > > these studies and that's due to simply the natural mean-sd
> > > relationship underlying the data-generating process for the raw scores
> > > (counts of some sort).
> > >
> > > If yes, I guess the difficult part (for me) becomes providing
> > > empirical support that at least there is evidence of mean-sd relation
> > > in my data whereby to provide evidence for the assumed the
> > > data-generating process. In my case (with longitudinal primary
> > > studies), for example, the smallest study gives the following results.
> > >
> > > Can I plot anything here (or for any other study) to empirically show
> > > mean-sd relation?
> > >
> > > Many thanks,
> > > Luke
> > >
> > > #----------------------------------------------
> > > d="
> > > Mt             SDt  nt       Mc     SDc nc time
> > > 0.0799 0.0367 21 0.0763 0.0389 26 0
> > > 0.113   0.0472 21 0.1095 0.0537 26 1"
> > >
> > > dat <- read.table(text=d, h=T)
> > > #----------------------------------------------
> > >
> > > On Sun, Oct 24, 2021 at 1:42 PM James Pustejovsky <jepusto using gmail.com> wrote:
> > >> Hi Luke,
> > >> My original response to your question was not specifically about mean-variance relationships within a study across multiple time points, but rather was more generally about mean-variance relationships in summary statistics. The simplest case here is where there's just one effect size per study and where the studies vary in the mean level of the outcome in a given group. I'll give you an example of the sort of data-generating process that I had in mind. It's important to bear in mind, though, that this is all entirely speculative---I'm *not* asserting that this model is appropriate for your data specifically, only that the example is one plausible situation where mean-variance relationships arise.
> > >> Suppose that we have k studies, each involving a two-group comparison, with groups of equal size. In study i, the outcomes in group A follow a poisson distribution with mean m_Ai, so that the variance of the outcomes in group A is also m_Ai, for i = 1,...,k. The outcomes in group B follow a poisson distribution with mean m_Bi, so the variance is also m_Bi. Now, suppose that there is a fixed, proportional relationship between pi_Bi and pi_Ai, so that pi_Bi = lambda pi_Ai for some lambda > 0. In other words, the treatment contrast is *constant* on the scale of the response ratio. However, the means in group A vary from study to study, according to some distribution, say a gamma distribution with shape parameter alpha and rate parameter beta.  What does this model imply about the distribution of standardized mean differences across this set of studies?
> > >> The SMD parameter for study i (call it delta_i) is the ratio of the mean difference to the square root of the pooled variance. So:
> > >> delta_i = (m_Bi - m_Ai) / sqrt[(m_Bi + m_Ai) / 2]
> > >> = (lambda - 1) m_Ai / sqrt[(1 + lambda) m_Ai / 2]
> > >> = sqrt(m_Ai) * (lambda - 1) * sqrt[2 * / (lambda + 1)]
> > >> The second and third term in the above expression are constants that only depend on the size of the response ratio. The first term is random because we have assumed that the group A means vary from study to study. It will therefore create heterogeneity in the SMD parameters---the greater the variance of the m_Ai's, the greater the heterogeneity in delta_i.
> > >> I've used poisson distributions for outcomes and the gamma distribution for the m_Ai's only for sake of simplicity. Something very similar would hold if we considered outcomes that were binomially distributed because the binomial distribution has variance that is strongly related to its mean. If the mean proportions vary from study to study, this implies that the variances will also vary from study-to-study in a non-linear way, which will induce heterogeneity in the SMD parameters.
> > >> James
> > >> On Sat, Oct 23, 2021 at 11:53 PM Luke Martinez <martinezlukerm using gmail.com> wrote:
> > >>> Hello All,
> > >>> I wanted to follow up on an answer
> > >>> (https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003354.html)
> > >>> on the list that, in a nutshell, says:
> > >>> Existence of a relationship between the Mean and SD for a given study
> > >>> group (e.g., Treatment), over a couple of time points, can potentially
> > >>> introduce additional heterogeneity in the SMD effect sizes across the
> > >>> studies.
> > >>> I was wondering how exactly this heterogeneity comes about?
> > >>> A part of me says that such a Mean-SD relationship for a given study
> > >>> group over time is indicative of what Hedges' (1981;
> > >>> https://doi.org/10.3102/10769986006002107) refers to as
> > >>> "subject-treatment interaction"?
> > >>> Another part of me says, no, "subject-treatment interaction" is
> > >>> controllable by adding a random effect for individuals, and thus it is
> > >>> different.
> > >>> I highly appreciate your insights regarding HOW Mean-SD relationship
> > >>> for study groups over time can potentially introduce additional
> > >>> heterogeneity in SMDs across studies?
> > >>> Luke
> > >>> PS:
> > >>> Here are the relation I see between mean and sd of a study group
> > >>> across 3 time points:
> > >>> group1_Ms = c(.39, .18, .13)
> > >>> group1_SDs = c(.25, .16, .13)
> > >>> plot(group1_Ms, group1_SDs, type="l")
> > >>> _______________________________________________
> > >>> R-sig-meta-analysis mailing list
> > >>> R-sig-meta-analysis using r-project.org
> > >>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis