[R-meta] Dear Wolfgang

[James Pustejovsky]

Hi JU,

This is a very interesting question. My personal sense of it is that
one should be very cautious when working with such highly imbalanced
datasets like this (where some studies contribute just a few effect
sizes while others contribute orders of magnitude more). The usual
methods for handling dependent effects (such as multi-level
meta-analysis, robust variance estimation, etc.) can do strange and
wacky things in this situation. As far as I know, their performance
has not been evaluated with data structures like this, and my
understanding is that the usual statistical theory supporting these
methods doesn't necessarily apply very well.

To make progress, I would recommend first carefully reviewing your
inclusion criteria and checking whether the effect sizes from the
agency studies are really comparable and aligned with the effect sizes
extracted from the peer-reviewed papers. One possible scenario is that
the peer-reviewed papers all only report outcomes on well-validated
scales, whereas the agency studies report outcomes on a kitchen sink
of different outcomes, including well-validated full scales but also
never-validated scales, sub-scales, single items, and assorted other
oddities. In this situation, I don't seem any benefit to throwing in
all the sub-scales and other chaff, just because it's available.
Better to use stricter inclusion criteria and just focus on the sound,
validated measures.

Another possible scenario is that the peer-reviewed studies report one
sort of information, whereas the agency studies report a categorically
different sort of information (with little or no overlap in terms of
the measures used, scale of the samples, etc.). In this case, it would
seem sensible to perform separate syntheses of the peer-reviewed
literature and the agency studies, then scratch your head over how to
understand differences between the two bodies of evidence.

Another possible scenario is that the peer-reviewed studies are all
poorly reported (and potentially selectively reported), whereas the
agency studies are not just completely reported, but also include
information on a wider range of relevant outcomes (e.g., effects over
longer follow-up times) than the peer-reviewed stuff. In that case, it
seems useful and potentially important to include the broader range of
effects from the agency studies. However, it needs to be done with
care. In particular, it will be critical to understand the sources of
variation *within* the agency studies. You might investigate this by
conducting preliminary analyses of the effects from each agency study
*on its own*, understanding what the important within-study moderators
are, and only then thinking about how to line up the evidence from the
agency studies with the evidence from the peer reviewed studies.

I'd be curious to hear more about the context you're working in, and
which (if any) of these scenarios you find yourself in. I'm also very
interested to hear others perspectives on this question.

Kind Regards,
James

> On Dec 9, 2020, at 12:07 PM, Ju Lee <juhyung2 using stanford.edu> wrote:
>
> Dear Wolfgang,

>
> I hope you are well these days.
>
> I had some general questions related to the data structure in mixed-effect models.
> We are currently working with data extracted from pee-reviewed papers as well as big data extracted from state or agency surveys.
> The issue we have is although we are including only 2-3 agency studies, each study can generate up to 1000-9000 effect sizes due to the abundance of data they produced.
>
> Conversely, the data collected from peer-reviewed articles are much smaller than that perhaps < 800 effect sizes combined. My co-authors want to use those agency data, but I am very concerned that including those data makes sense.
>
> So the question is:
>
>  1.  Is it even reasonable to consider including such few studies that will pretty much dominate the entire data?
>  2.  Can common mixed-effect model approaches with study random factor account for such disproportionate contribution of few studies?
>
> It would be extremely helpful to hear your perspectives.
> Thank you
>
> Best,
> JU
>
>    [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis