[R-meta] Dear Wolfgang

[Ju Lee]

Dear James,

Thank you so much for these detailed thoughts and suggestions. My co-authors and I find this input extremely helpful.

Currently, we are analyzing fishery data across the US, trying to understand how habitat or environmental contexts influence fish productivity by combining both peer-reviewed and agency data. We think our current issue is most related to your first point. We have >80 peer-reviewed papers and 3-4 agency dataset.

One major difference between peer-reviewed articles and agency data is that agency data are collected more frequently, across larger areas and different sites, and over a longer time frame (> 20 yr long data accumulated). Also, they do indiscriminately record all catch, whereas peer-reviewed papers often report ones that are more relevant to their research question (one that is more aligned with our research question as well).

So, after reading your comments, I envision us applying a stricter rule for the agency data. We have 1000 or so effect sizes combining many peer-reviewed articles, whereas each dataset generates between a whopping 9000-13000 effect sizes if we apply the same inclusion criteria.

Our follow-up questions are:

1. Given that we think there is still some value in including the agency data in our analysis, is it reasonable to apply different criteria or rules (more strict) just for the agency data?

For example, in our peer-reviewed data, we treat samplings conducted in different areas and years as independent studies and as these effects are matched with discrete measurements for our moderator of interest (say temperature or depth). I am wondering if it is justifiable if we apply a different protocol just for agency data (ex. merging and pooling across all years and sites and just generate a single effect size for each fish species OR only including randomly chosen year or site from the dataset) for the sake of not taking over the entire data with agency ones.

2. One option we were considering was to run our models with and without agency data and report both. However, you pointed out that that model output (including such an abnormally large study) may not be reliable at all if there is such a huge study size differences to begin with. So, my understanding is that this should not be one of our options unless we can significantly reduce the number of agency data being incorporated?

3. Finally, you mentioned analyzing the agency data separately. I have not considered this option but say we have two agency datasets that we combine to run our models. There would be plenty of effect sizes (coming from different sites, years, observers, and fish species) but only two levels of study (data 1, 2). I am unsure if this approach would make any sense, but do you have any additional thoughts on the validity of such models or approaches?

Thank you very much for your time, James. I hope my answers and follow-up questions are clear enough.
Best regards,

JU
________________________________
From: James Pustejovsky <jepusto using gmail.com>
Sent: Wednesday, December 9, 2020 6:12 PM
To: Ju Lee <juhyung2 using stanford.edu>
Cc: Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>; R meta <r-sig-meta-analysis using r-project.org>
Subject: Re: [R-meta] Dear Wolfgang

Hi JU,

This is a very interesting question. My personal sense of it is that
one should be very cautious when working with such highly imbalanced
datasets like this (where some studies contribute just a few effect
sizes while others contribute orders of magnitude more). The usual
methods for handling dependent effects (such as multi-level
meta-analysis, robust variance estimation, etc.) can do strange and
wacky things in this situation. As far as I know, their performance
has not been evaluated with data structures like this, and my
understanding is that the usual statistical theory supporting these
methods doesn't necessarily apply very well.

To make progress, I would recommend first carefully reviewing your
inclusion criteria and checking whether the effect sizes from the
agency studies are really comparable and aligned with the effect sizes
extracted from the peer-reviewed papers. One possible scenario is that
the peer-reviewed papers all only report outcomes on well-validated
scales, whereas the agency studies report outcomes on a kitchen sink
of different outcomes, including well-validated full scales but also
never-validated scales, sub-scales, single items, and assorted other
oddities. In this situation, I don't seem any benefit to throwing in
all the sub-scales and other chaff, just because it's available.
Better to use stricter inclusion criteria and just focus on the sound,
validated measures.

Another possible scenario is that the peer-reviewed studies report one
sort of information, whereas the agency studies report a categorically
different sort of information (with little or no overlap in terms of
the measures used, scale of the samples, etc.). In this case, it would
seem sensible to perform separate syntheses of the peer-reviewed
literature and the agency studies, then scratch your head over how to
understand differences between the two bodies of evidence.

Another possible scenario is that the peer-reviewed studies are all
poorly reported (and potentially selectively reported), whereas the
agency studies are not just completely reported, but also include
information on a wider range of relevant outcomes (e.g., effects over
longer follow-up times) than the peer-reviewed stuff. In that case, it
seems useful and potentially important to include the broader range of
effects from the agency studies. However, it needs to be done with
care. In particular, it will be critical to understand the sources of
variation *within* the agency studies. You might investigate this by
conducting preliminary analyses of the effects from each agency study
*on its own*, understanding what the important within-study moderators
are, and only then thinking about how to line up the evidence from the
agency studies with the evidence from the peer reviewed studies.

I'd be curious to hear more about the context you're working in, and
which (if any) of these scenarios you find yourself in. I'm also very
interested to hear others perspectives on this question.

Kind Regards,
James

> On Dec 9, 2020, at 12:07 PM, Ju Lee <juhyung2 using stanford.edu> wrote:
>
> Dear Wolfgang,

>
> I hope you are well these days.
>
> I had some general questions related to the data structure in mixed-effect models.
> We are currently working with data extracted from pee-reviewed papers as well as big data extracted from state or agency surveys.
> The issue we have is although we are including only 2-3 agency studies, each study can generate up to 1000-9000 effect sizes due to the abundance of data they produced.
>
> Conversely, the data collected from peer-reviewed articles are much smaller than that perhaps < 800 effect sizes combined. My co-authors want to use those agency data, but I am very concerned that including those data makes sense.
>
> So the question is:
>
>  1.  Is it even reasonable to consider including such few studies that will pretty much dominate the entire data?
>  2.  Can common mixed-effect model approaches with study random factor account for such disproportionate contribution of few studies?
>
> It would be extremely helpful to hear your perspectives.
> Thank you
>
> Best,
> JU
>
>    [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

	[[alternative HTML version deleted]]