Hey Andrew, grateful for that response. I think you're right, and well, I hope you're right. Thanks also to David for the excellent idea (below.)

On the rating of emotions, there are two common practices in research similar to what I described (non-expert appraisals).  If raw scores show good correlations ("good" defined variously) you take the mean of the ratings (for each item).  Alternatively, if there isn't "good" agreement, sometimes researchers will ask raters to discuss their disagreements, until they agree. Hmmm...   

Over the weekend, I found an interesting article that addresses some of these problems, though in the context of orangutan personality.  I contacted the first author, Mark Adams, who offered some helpful suggestions below. The orangutan article: 

DOI 10.1007/s10519-012-9537-y

[Adams et  al., 2012]	Adams, M. J., King, J. E., and Weiss, A. (2012). The majority of genetic variation in orangutan personality and subjective well-being is nonadditive. Behavior genetics, 42(4):675–686.

@article{adams2012majority,
	Author = {Adams, Mark James and King, James E and Weiss, Alexander},
	Journal = {Behavior genetics},
	Number = {4},
	Pages = {675--686},
	Publisher = {Springer},
	Title = {The majority of genetic variation in orangutan personality and subjective well-being is nonadditive},
	Volume = {42},
	Year = {2012}}


Mark James Adams <mark.adams@sheffield.ac.uk>
To: Joseph Bulbulia <joseph.bulbulia@icloud.com>


Hi Joseph

Glad you enjoyed the orang paper.

> 1. Estimating variances for individual raters.
> How exactly did you build models for each rater's judgment of
> personality? 

> 2. Modelling variance of raters as distinct from orangutans
> How did you then model the variances of individual orangutans (V_id) as
> distinct from the variances of raters (V_j). You mentioned that you used
> orangutan and rater IDs as predictors

Yes, both are included as random effects. In my case each personality domain was made up of multiple items, so my data looked something like (I'll use extraversion and agreeableness, with two items each, as example)

id	rater	item	extra	agree
1	a	soc	3	NA
1	a	frd	4	NA
1	a	gnt	NA	1
1	a	help	NA	2
1	b	soc	4	NA
1	b	frd     4	NA
...
2	a	soc	7	NA

The model looked something like

MCMCglmm(cbind(extra, agree) ~ trait + trait:age + trait:sex + trait:age:sex,
        random=~ us(trait):id + us(trait):rater,
        rcov=~idh(trait):units
)

Because extraversion and agreeableness item scores are not observed 'together' in the data frame, the residual covariance is fixed at 0 (with the residual variances fixed at 1 in the prior). In your case, you can estimate the residual covariances. The way to estimate the covariances but fix the variances to 1 using a cor() structure.

First, don't fix the prior for the residual

[[JB my error. I think I misread advice for fixing ordinal R structures on p.23 of the 2010 MCMCglmm vignette.)]]


prior.fw.0 = list(
 R = list(V =diag(2), n=1.002),

(you'll need to choose something sensible for the degree-of-belief depending on how much data you have)

Then the model part is

MCMCglmm(...,
	rcov= ~ cor(trait):units,
)

> 
> 3. ICC
> You worked both on the raw and latent data scales, which seems
> appropriate given that your data were ordinal. I face a similar issue
> here with my ordinal rater data.

Deciding which variance components to use depends on the question. The thing to remember is that the ICC is the expected correlation between outcomes of a particular group member (level of a random effect), averaging over one or more of the other random effects (AKA the repeatability).

For the orangutans, I was interested in a rater's average assessment of an orang-utan, so I wanted to remove the residual (subject * rater *item variance) and distribution specific variance (which is 1 for ordinal), to get the latent scale repeatability

with(model, VCV[,'extra:extra.rater'] / (rowSums(VCV[,c('extra:extra.id', 'extra:extra.rater'))

If I wanted to calculate the correlation between rater's assessment on a given item with their rating of another randomly chosen item and orang-utan, I could add in the residual and ordinal variance

with(model, VCV[,'extra:extra.rater'] / (rowSums(VCV[,c('extra:extra.id', 'extra:extra.rater', 'extra:extra.units')]) + 1))

Once you do this you can also look at the influence the raters have on the correlation between arousal and valence by fitting `us(trait):rater`. Hope that makes sense.

A few other comments about your model:

[snip: comment on fitting main effects + interactions (response: long story: there is disagreement among my collaborators about the structure of the quasi-experiment, i.e. whether a 2 x 5 x 2 factorial design captures it, to be settled later...)]


(2) The syntax to fit random slopes for the participants should be: us(trait + trait:phase):id


On 19/04/2013, at 12:57 PM, Andrew Robinson <A.Robinson@ms.unimelb.edu.au> wrote:
> Hi Joseph,
> 
> thanks for this detailed summary.  Based on my understanding, I think
> that it is defensible to include the raters as random effects in the
> model, and I think that doing so provides a more faithful
> representation of your experimental design than would excluding them.
> 
> Definitely not nuts.
> 
> On the niggles: I'm not sure what exactly you mean by "averaging over
> the ratings".  It sounds risky to me.
> 
> Cheers
> 
> Andrew
> 
> On Tue, Apr 16, 2013 at 7:50 PM, Joseph Bulbulia
> <joseph.bulbulia@icloud.com> wrote:
>> Hi all,
>> 
>> Two of you asked for more information.  Sorry for the long-winded account, written in haste.
>> 
>> THE QUASI EXPERIMENT
>> * The fire-walking ritual consisted of a series of 26 ordeals by fire.
>> * Each fire-walker traversed a burning bed of coals (677 Celsius -- I actually measured it with a pyrometer. Such instruments exit!).
>> * In sixteen of these events, fire-walkers were carrying a passenger.
>> * Total duration of each fire-walk = < 5 seconds, which we carved up into five phases.
>> 
>> I constructed a make-shift plot plan of the ritual here (following ideas in Walter Stroup's recent book on GLMMs).  Not sure I'm happy with it, but it will give you the gist.
>> 
>> https://www.dropbox.com/s/0o0a9kkrh5ttsd2/plot.plan.emotions_firewalk.pdf
>> Hypotheses 1:
>> Anthropologists have long maintained that rituals cause a melding of emotions, what they call "collective effervescence". You've felt this surely.  Being connected with others at a big event … This "merge" model predicts that arousal and valence will tend to be coupled among participants, irrespective of ritual roles.  (In another paper, we demonstrated heart rhythm coupling among fire walkers and observers; the study was published in 2011.)
>> 
>> Hypothesis 2:
>> Another anthropological tradition predicts differentiation in emotions depending on ritual role.  This "verge" model predicts that ritual participants who undertake a rite of passage will express different emotions. Think of a PhD thesis defence. The candidate's ordeal is the inquisitor's delight! (In our heart rhythm paper we also found differences in synchrony which were predicted by ritual role and social distance. This study is just a follow up using another biomarker.)
>> 
>> To assess whether emotional dynamics merge or verge, we sampled images for each ritual participant (n=42, 26 firewalkers and 16 passengers) at five different phases of the fire-walk.
>> 
>> There's evidence of cultural variability in emotions, so the images were independently rated by four judges from the part of Spain, roughly from the area where the ritual happened.
>> 
>> (Note: if I could do this over again, I'd get more raters, but this sort of number is typical in psychological research: it is probably OK for the task a hand, which does not require exact estimates, only rough assessments of trends among each ritual group).
>> 
>> See how you do here.  Merge or Verge?
>> https://www.dropbox.com/s/xmbci5814h73i0l/4_MF_e5.png
>> 
>> Images were rated for “valence” and “arousal” on Likert scales from 1-7.  I didn't run an ICC because I wasn't sure whether this is appropriate for ordinal data (If anyone knows I'd be grateful, but I didn't want to bog down the list with too many questions).  Kendall's coefficient of concordance was 0.513.   As is typical in emotions research, then, judgements were not all that concordant.  But again noisy signals are ok in the context of this study.  There's a larger philosophical discussion about whether emotions are intrinsically vague and context-dependent creatures.  We can set that to the side though.  Crude signals, in this case, are fine.
>> 
>> 
>> The Model
>>  Fixed effects for Phase x Role strongly improve the intercept only model, and show merge for arousal and verge for valence. This finding is supported in all other models.
>>  Random slopes for participants by Phase do better than random intercepts and slopes.
>>  Random effects for Events improve the model, but there's no improvement by including effects for Dyadic pairs.
>>  I used an ordinal family because the data are ordinal.
>>  I fixed the R variance to 1 because this is what Jarrod Hadfield's Course Notes recommend, and he is a man who knows what he's talking about.
>> 
>> Key point
>> Nothing hangs on putting raters into the model!!   The outcome remains the same with respect to the hypotheses.  I could leave them out (and probably will).  However it seems to me that the raters are somehow part of the effect, in a way that is very roughly analogous to meta-analysis studies. (However I did not attempt MEV… seemed a bit extreme, but who knows?!!)
>> 
>> 
>> Other niggles
>> My psychologist collaborator (experienced with LMMs using HLM and MPLUS)  suggested averaging over the ratings. This is standard practice in psychology. In fact, psychologist do this all the time wherever they have highly correlated measures for the same trait (e.g. personality).
>> This strikes me as OK for most purposes, but it is also odd, because you loose a signal for the variance of your measures.
>> 
>> Again, sorry for stealing time.  Thanks for any help.
>> 
>> 
>> 
>> 
>> 
>> On 16/04/2013, at 8:05 PM, David Duffy <David.Duffy@qimr.edu.au> wrote:
>> 
>>> On Tue, 16 Apr 2013, Joseph Bulbulia wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I?d like to model emotional dynamics in a highly arousing firewalk ritual.
>>>> 
>>>> Four judges rated images from 42 participants for arousal and valence. The predictor variables are ritual ?phase? and ?role.?
>>>> 
>>>> Question 1
>>>> Any thoughts about how best to handle the rater assessments?
>>>> 
>>>> Specifically, is it nuts to explicitly include a component for raters in the random component of the model?
>>> 
>>> So what do the inter-rater agreements look like?  I presume rater is actually a nuisance variable?  The path model I would usually use would have phase and role acting on the averaged-over-raters a and v scores (measurement model bit).
>>> 
>>> Just 2c, David Duffy.
>> 
>> 
>>        [[alternative HTML version deleted]]
>> 
>> 
>> _______________________________________________
>> R-sig-mixed-models@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>> 
> 
> 
> 
> -- 
> Andrew Robinson
> Deputy Director, ACERA
> Senior Lecturer in Applied Statistics                      Tel: +61-3-8344-6410
> Department of Mathematics and Statistics            Fax: +61-3-8344 4599
> University of Melbourne, VIC 3010 Australia
> Email: a.robinson@ms.unimelb.edu.au    Website: http://www.ms.unimelb.edu.au
> 
> FAwR: http://www.ms.unimelb.edu.au/~andrewpr/FAwR/
> SPuR: http://www.ms.unimelb.edu.au/spuRs/


	[[alternative HTML version deleted]]

