[R-sig-ME] [FORGED] Re: Using variance components of lmer for ICC computation in reliability study

[Doran, Harold]

This seems to me like the tail is wagging the dog. I don't think the question should be framed around, "what can I do to get an ICC?" The question should be around what distributional family is appropriate given the observed data?

If an ICC cannot be made available after the appropriate link function is chosen, then I'm not sure that's really of any consequence. The point of a random effects model is *not* to yield an ICC. But, to yield appropriate estimates for the fixed effects and marginal variances.

If these data are treated as ordinal, that's a LOT of threshold parameters to estimate and might be quite expensive to compute. 


-----Original Message-----
From: Pierre de Villemereuil [mailto:pierre.de.villemereuil using mailoo.org] 
Sent: Friday, June 15, 2018 11:05 AM
To: r-sig-mixed-models using r-project.org
Cc: Ben Bolker <bbolker using gmail.com>; Doran, Harold <HDoran using air.org>; Bernard Liew <B.Liew using bham.ac.uk>
Subject: Re: [R-sig-ME] [FORGED] Re: Using variance components of lmer for ICC computation in reliability study

Hi,

> However, I'm not sure how one would go about computing an ICC from 
> ordinal data

I've never used the package "ordinal", but if it's anything like the "ordinal" family of MCMCglmm, then computing an ICC on the liability scale would be fairly easy, as one would just proceed as always and simply add the so-called "link variance" corresponding to the chosen link function (1 for probit, (pi^2)/3 for logit). E.g. for a given variance component Vcomp and a probit link:
ICC = Vcomp / (sum(variance components of the model) + 1)

However, computing an ICC on the data scale would be much more difficult as it is actually multivariate...

I think in the case when such scores were used, having the estimate on the liability scale make sense though, as the binning is more due to our inability of finely measuring this scale rather than an actual property of the system.

Cheers,
Pierre.

Le vendredi 15 juin 2018, 03:27:54 CEST Ben Bolker a écrit :
> More generally, the best way to fit this kind of model is to use an
> *ordinal* model, which assumes the responses are in increasing 
> sequence but does not assume the distance between levels (e.g. 1 vs 2,
> 2 vs 3 ...) is uniform.  However, I'm not sure how one would go about 
> computing an ICC from ordinal data ... (the 'ordinal' package is the 
> place to look for the model-fitting procedures). Googling it finds 
> some stuff, but it seems that it doesn't necessarily apply to complex 
> designs ...
> 
> https://stats.stackexchange.com/questions/3539/inter-rater-reliability
> -for-ordinal-or-interval-data 
> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/
> 
> 
> On Thu, Jun 14, 2018 at 6:58 PM, Doran, Harold <HDoran using air.org> wrote:
> > That’s a helpful clarification, Rolf. However, with gaussian normal 
> > errors in the linear model, we can’t *really* assume they would 
> > asymptote at 1 or 10. My suspicion is that these are likert-style 
> > ordered counts of some form, although the OP should clarify. In 
> > which case, the 1 or 10 are limits with censoring, as true values 
> > for some measured trait could exist outside those boundaries (and I 
> > suspect the model is forming predicted values outside of 1 or 10).
> >
> >
> >
> > On 6/14/18, 6:33 PM, "Rolf Turner" <r.turner using auckland.ac.nz> wrote:
> >
> >>
> >>On 15/06/18 05:35, Doran, Harold wrote:
> >>
> >>> Well no, you¹re specification is not right because your variable 
> >>> is not continuous as you note. Continuous means it is a real 
> >>> number between -Inf/Inf and you have boundaries between 1 and 10. 
> >>> So, you should not be using a linear model assuming the outcome is continuous.
> >>
> >>I think that the foregoing is a bit misleading.  For a variable to 
> >>be continuous it is not necessary for it to have a range from 
> >>-infinity to infinity.
> >>
> >>The OP says that dv  "is a continuous variable (scale 1-10)".  It is 
> >>not clear to me what this means.  The "obvious"/usual meaning or 
> >>interpretation would be that dv can take (only) the (positive 
> >>integer) values 1, 2, ..., 10.  If this is so, then a continuous 
> >>model is not appropriate.  (It should be noted however that people 
> >>in the social sciences do this sort of thing --- i.e. treat discrete 
> >>variables as continuous --- all the time.)
> >>
> >>It is *possible* that dv can take values in the real interval 
> >>[1,10], in which case it *is* continuous, and a "continuous model" 
> >>is indeed appropriate.
> >>
> >>The OP should clarify what the situation actually is.
> >>
> >>cheers,
> >>
> >>Rolf Turner
> >>
> >>--
> >>Technical Editor ANZJS
> >>Department of Statistics
> >>University of Auckland
> >>Phone: +64-9-373-7599 ext. 88276
> >>
> >>> On 6/14/18, 11:16 AM, "Bernard Liew" <B.Liew using bham.ac.uk> wrote:
> >>>
> >>>> Dear Community,
> >>>>
> >>>>
> >>>> I am doing a reliability study, using the methods of 
> >>>> https://www.ncbi.nlm.nih.gov/pubmed/28505546. I have a question 
> >>>> on the lmer formulation and the use of the variance components.
> >>>>
> >>>> Background: I have 20 subjects, 2 fixed raters, 2 testing 
> >>>> sessions, and
> >>>> 10 trials per sessions. my dependent variable is a continuous 
> >>>> variable (scale 1-10). Sessions are nested within each 
> >>>> subject-assessor combination. I desire a ICC (3) formulation of 
> >>>> inter-rater and inter-session reliability from the variance components.
> >>>>
> >>>> My lmer model is:
> >>>>
> >>>> lmer (dv ~ rater + (1|subj) + (1|subj:session), data = df)
> >>>>
> >>>> Question:
> >>>>
> >>>>   1.  is the model formulation right? and is my interpretation of 
> >>>>the  variance components for ICC below right?
> >>>>   2.  inter-rater ICC = var (subj) / (var(subj) + var (residual)) 
> >>>># I  read that the variation of raters will be lumped with the residual
> >>>>   3.  inter-session ICC =( var (subj) + var (residual)) /( var 
> >>>>(subj) +  var (subj:session) + var (residual))  some simulated 
> >>>>data:
> >>>> df = expand.grid(subj = c(1:20), rater = c(1:2), session = 
> >>>>c(1:2), trial  = c(1:10))  df$vas = rnorm (nrow (df_sim), mean = 
> >>>>3, sd = 1.5)
> >>>>
> >>>> I appreciate the kind response.
> >>
> >>
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list 
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> 
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list 
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models