[R-sig-ME] How to deal with outcomes assessed by raters?

[Andrew Robinson]

Hi Joseph,

thanks for this detailed summary.  Based on my understanding, I think
that it is defensible to include the raters as random effects in the
model, and I think that doing so provides a more faithful
representation of your experimental design than would excluding them.

Definitely not nuts.

On the niggles: I'm not sure what exactly you mean by "averaging over
the ratings".  It sounds risky to me.

Cheers

Andrew

On Tue, Apr 16, 2013 at 7:50 PM, Joseph Bulbulia
<joseph.bulbulia at icloud.com> wrote:
> Hi all,
>
> Two of you asked for more information.  Sorry for the long-winded account, written in haste.
>
> THE QUASI EXPERIMENT
> * The fire-walking ritual consisted of a series of 26 ordeals by fire.
> * Each fire-walker traversed a burning bed of coals (677 Celsius -- I actually measured it with a pyrometer. Such instruments exit!).
> * In sixteen of these events, fire-walkers were carrying a passenger.
> * Total duration of each fire-walk = < 5 seconds, which we carved up into five phases.
>
> I constructed a make-shift plot plan of the ritual here (following ideas in Walter Stroup's recent book on GLMMs).  Not sure I'm happy with it, but it will give you the gist.
>
> https://www.dropbox.com/s/0o0a9kkrh5ttsd2/plot.plan.emotions_firewalk.pdf
> Hypotheses 1:
> Anthropologists have long maintained that rituals cause a melding of emotions, what they call "collective effervescence". You've felt this surely.  Being connected with others at a big event … This "merge" model predicts that arousal and valence will tend to be coupled among participants, irrespective of ritual roles.  (In another paper, we demonstrated heart rhythm coupling among fire walkers and observers; the study was published in 2011.)
>
> Hypothesis 2:
> Another anthropological tradition predicts differentiation in emotions depending on ritual role.  This "verge" model predicts that ritual participants who undertake a rite of passage will express different emotions. Think of a PhD thesis defence. The candidate's ordeal is the inquisitor's delight! (In our heart rhythm paper we also found differences in synchrony which were predicted by ritual role and social distance. This study is just a follow up using another biomarker.)
>
> To assess whether emotional dynamics merge or verge, we sampled images for each ritual participant (n=42, 26 firewalkers and 16 passengers) at five different phases of the fire-walk.
>
> There's evidence of cultural variability in emotions, so the images were independently rated by four judges from the part of Spain, roughly from the area where the ritual happened.
>
> (Note: if I could do this over again, I'd get more raters, but this sort of number is typical in psychological research: it is probably OK for the task a hand, which does not require exact estimates, only rough assessments of trends among each ritual group).
>
> See how you do here.  Merge or Verge?
> https://www.dropbox.com/s/xmbci5814h73i0l/4_MF_e5.png
>
> Images were rated for “valence” and “arousal” on Likert scales from 1-7.  I didn't run an ICC because I wasn't sure whether this is appropriate for ordinal data (If anyone knows I'd be grateful, but I didn't want to bog down the list with too many questions).  Kendall's coefficient of concordance was 0.513.   As is typical in emotions research, then, judgements were not all that concordant.  But again noisy signals are ok in the context of this study.  There's a larger philosophical discussion about whether emotions are intrinsically vague and context-dependent creatures.  We can set that to the side though.  Crude signals, in this case, are fine.
>
>
> The Model
>   Fixed effects for Phase x Role strongly improve the intercept only model, and show merge for arousal and verge for valence. This finding is supported in all other models.
>   Random slopes for participants by Phase do better than random intercepts and slopes.
>   Random effects for Events improve the model, but there's no improvement by including effects for Dyadic pairs.
>   I used an ordinal family because the data are ordinal.
>   I fixed the R variance to 1 because this is what Jarrod Hadfield's Course Notes recommend, and he is a man who knows what he's talking about.
>
> Key point
> Nothing hangs on putting raters into the model!!   The outcome remains the same with respect to the hypotheses.  I could leave them out (and probably will).  However it seems to me that the raters are somehow part of the effect, in a way that is very roughly analogous to meta-analysis studies. (However I did not attempt MEV… seemed a bit extreme, but who knows?!!)
>
>
> Other niggles
> My psychologist collaborator (experienced with LMMs using HLM and MPLUS)  suggested averaging over the ratings. This is standard practice in psychology. In fact, psychologist do this all the time wherever they have highly correlated measures for the same trait (e.g. personality).
> This strikes me as OK for most purposes, but it is also odd, because you loose a signal for the variance of your measures.
>
> Again, sorry for stealing time.  Thanks for any help.
>
>
>
>
>
> On 16/04/2013, at 8:05 PM, David Duffy <David.Duffy at qimr.edu.au> wrote:
>
>> On Tue, 16 Apr 2013, Joseph Bulbulia wrote:
>>
>>> Hi all,
>>>
>>> I?d like to model emotional dynamics in a highly arousing firewalk ritual.
>>>
>>> Four judges rated images from 42 participants for arousal and valence. The predictor variables are ritual ?phase? and ?role.?
>>>
>>> Question 1
>>> Any thoughts about how best to handle the rater assessments?
>>>
>>> Specifically, is it nuts to explicitly include a component for raters in the random component of the model?
>>
>> So what do the inter-rater agreements look like?  I presume rater is actually a nuisance variable?  The path model I would usually use would have phase and role acting on the averaged-over-raters a and v scores (measurement model bit).
>>
>> Just 2c, David Duffy.
>
>
>         [[alternative HTML version deleted]]
>
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>



-- 
Andrew Robinson
Deputy Director, ACERA
Senior Lecturer in Applied Statistics                      Tel: +61-3-8344-6410
Department of Mathematics and Statistics            Fax: +61-3-8344 4599
University of Melbourne, VIC 3010 Australia
Email: a.robinson at ms.unimelb.edu.au    Website: http://www.ms.unimelb.edu.au

FAwR: http://www.ms.unimelb.edu.au/~andrewpr/FAwR/
SPuR: http://www.ms.unimelb.edu.au/spuRs/