[R-meta] MA of regression coefficients (as accuracy indicators) in pre-post test

[James Pustejovsky]

Unfortunately I don’t think there is a way to calculate Cb (or the ccc) without knowing the mean of one of the variables. Without one or the other, I can’t see any way to determine the degree of shift (above or below the line y = x) in the regression line.

> On Dec 1, 2017, at 1:25 PM, Emerson M. Del Ponte <edelponte at gmail.com> wrote:
> 
> James,
> 1) Yes, and I have the Lin’s CCC for about 15% of the articles for which I have the raw data, or all estimates for the sample of leaves for each rater.
> 2) No, what I have is only the regression coefficients and correlation coefficient (Pearson’s r)  (estimates regressed again actual values) for each rater. Wonder if there is a way to calculate the Lin’s bias coefficient (Cb) from the intercept and slope? The Lin’s CCC is r * Cb, and I have only r.
> Thanks,
> Emerson
> 
> Emerson,
> 
> Two quick questions to see if there is an easy solution:
> 1) Do you mean Lin's concordance correlation (DOI: 10.2307/2532051)?
> 2) Do you have (or can you extract) the means of the pre-test or post-test
> values for each rater?
> If the answers to both questions are "yes" then it is possible to compute
> an estimate of the concordance correlation from the available summary
> statistics.
> 
> James
> 
> On Fri, Dec 1, 2017 at 6:06 AM, Emerson M. Del Ponte <edelponte at gmail.com <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>>
> wrote:
> 
>> Dear All,
>> 
>> I have collected data from primary studies where an assessment aid
>> (diagrams) was developed and tested for improvements in accuracy and
>> precision of visual assessments (% leaf area affected - disease symptoms)
>> compared with unaided ones.
>> 
>> A  sample of raters was used in each study (varied across studies). Each
>> rater made two assessments (estimates of % leaf affected area) for a same
>> sample of leaves (a range of values from 0 to 100%), first unaided (und)
>> and then aided (aid) (pre-post test).
>> 
>> Visual estimates from each assessment (e.g. 50 ratings) were regressed
>> against “actual” values. Regression coefficients (beta1 and beta1) are
>> measures of accuracy and the correlation coefficient (r) is a measure of
>> precision. So, I have three variables for each rater.
>> 
>> The data look like this:
>> 
>> Study Rater     r_und   r_aid
>> 1               1       0.65            0.76
>> 1               2       0.76            0.90
>> 1               3       0.80            0.90
>> .               .       .               .
>> 
>> Wolfgang has kindly helped me (more than two years ago!) to preliminary
>> fit a multi-level model in metafor to summarize the gains precision
>> (correlation coefficients). The effect-size was the absolute difference
>> (r_aided - r_unaided) and there was a way to calculate sampling variance.
>> 
>> This worked fine, but I have been struggling to define what would be the
>> appropriate approach to analyze gains in accuracy using beta1 and beta1. In
>> a primary study, vote-couting was used to infer on the value of the aid
>> based on number of raters with significant (P = 0.05) departures of beta1
>> and beta2 from 0 and 1, respectively, both unaided and aided.
>> 
>> My problem is that I cannot calculate an index (such as concordance
>> correlation) for accuracy because raw data was not available (I have the
>> correlation coefficient, which explains in part the overall accuracy or
>> concordance). So, I don’t think that an estimate of absolute difference in
>> beta1 and beta2 between aided and unaided estimate for each rater make
>> sense? I can see (histograms) that, in general, b1 and b2 are closer to 0
>> and 1, respectively when using the aid.
>> 
>> What I did so far was to aggregate (means) each coefficients by study to
>> then obtain the sampling variance (raters as samples) for the study. Then,
>> a bivariate model was fitted to these data from k studies and the estimates
>> of b1 and b2 were obtained for each condition. The data looks exactly like
>> shown above, but with b1_und and b1_aid, etc.
>> 
>> I am not sure if there is a better way to analyze these data. Ideally, I
>> would like to be able to calculate the concordance coefficient (which
>> includes the correlation coefficient) but this seems not possible without
>> the raw data.  What I have allows me to analyze separately precision and
>> accuracy, but while the former seems OK (I have an effect-size and sampling
>> variance for each rater), I am not sure how to better estimate gains in
>> accuracy using the regression coefficients.
>> 
>> Any thoughts or examples on this?
>> 
>> Thanks,
>> 
>> Emerson
>> 
>> Prof. Emerson M. Del Ponte
>> Departamento de Fitopatologia
>> Universidade Federal de Viçosa
>> Viçosa, MG - Brasil
>> +55 (31) 3899-1103 <+55%2031%203899-1103>
>> Twitter: @edelponte
>> 
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis at r-project.org <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
> 
>    [[alternative HTML version deleted]]
> 
> 
>    [[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis