[R-sig-ME] Accounting for dv score's validity using weights - metafor or lme4?

[Viechtbauer Wolfgang (SP)]

Hi Isaac,

If I understand you correctly, the data structure looks something like this:

participant response coder value
================================
1           1        1     y
1           1        2     y
...         ...     ...    ...
1           1       25     y
--------------------------------
1           2        1     y
1           2        2     y
...         ...     ...    ...
1           2       25     y
--------------------------------
2           1        1     y
2           1        2     y
...         ...     ...    ...
2           1       25     y
--------------------------------
2           2        1     y
2           2        2     y
...         ...     ...    ...
2           2       25     y
--------------------------------

So, subject 1 gave two responses, each of which was coded by 25 coders; subject 2 gave 2 (or maybe more) responses, again each of which was coded by 25 coders; and so on. I assume the same coders were used across participants (with maybe some slight variation) and that you know who the coders were (so coder 1 for subject 1 is the same person as coder 1 for subject 2, since the coder ID numbers are the same).

I see no need for treating this 'meta-analytically'. You have the raw data, so you might as well analyze them as such. A starting point to consider would be a model along the lines of:

lme(y ~ 1 + (1 | participant/response) + (1 | coder), ...)

that is, random effects for participants, random effects for responses within participants, and then crossed random efects for coders (since coders may have a propensity to give higher/lower values, regardless of whose participant's responses they are coding).

Since the outcome represents a probability, you might run into floor/ceiling issues, leading to violations of distributional assumptions (normality of random effects, normality of residuals), so this is something to examine.

Best,
Wolfgang

-- 
Wolfgang Viechtbauer, Ph.D., Statistician | Department of Psychiatry and 
Neuropsychology | Maastricht University | P.O. Box 616 (VIJV1) | 6200 MD 
Maastricht, The Netherlands | +31 (43) 388-4170 | http://www.wvbauer.com 

>-----Original Message-----
>From: R-sig-mixed-models [mailto:r-sig-mixed-models-bounces at r-
>project.org] On Behalf Of Isaac Fradkin
>Sent: Saturday, 20 January, 2018 12:35
>To: r-sig-mixed-models at r-project.org
>Subject: [R-sig-ME] Accounting for dv score's validity using weights -
>metafor or lme4?
>
>Dear all.
>
>I have a dataset of 964 data points clustered within 250 participant.
>Each participant gave several open-responses to a specific question, and
>these responses were then coded by additional participants (~25 per
>response) on a scale from 0-100 (representing probability, but it doesn't
>seem to matter for the question here). Now, the simplest way to analyze
>this is to simply take the median (or winsorized mean) of the different
>ratings, and using this as the dv. However, because some responses
>produced more disagreement between raters than others, I thought it might
>be wise to account for this 'uncertainty' in some way.
>I've considered two ways. First, I believe one might try to think of the
>problem in a meta-analytic framework. Each median rating in-fact
>represents sort of a 'sample estimate' taken from a distribution which
>variance corresponds to the variance between ratings, just like in random
>effects meta-analysis each effect size is weighted by its precision
>because it is assumed to come from a population effect size for this
>study. If this sound ok I could use the metafor package to specify a
>multilevel model. The problem is that I don't know of any study using
>such an approach.
>Second, I know there is a weights argument in lme4, but not exactly sure
>how it works, and how its results will relate to the first solution.
>
>Any advise or relevant reference is highly appreciated!
>Isaac.