[R-meta] MA of regression coefficients (as accuracy indicators) in pre-post test

[James Pustejovsky]

Emerson,

Two quick questions to see if there is an easy solution:
1) Do you mean Lin's concordance correlation (DOI: 10.2307/2532051)?
2) Do you have (or can you extract) the means of the pre-test or post-test
values for each rater?
If the answers to both questions are "yes" then it is possible to compute
an estimate of the concordance correlation from the available summary
statistics.

James

On Fri, Dec 1, 2017 at 6:06 AM, Emerson M. Del Ponte <edelponte at gmail.com>
wrote:

> Dear All,
>
> I have collected data from primary studies where an assessment aid
> (diagrams) was developed and tested for improvements in accuracy and
> precision of visual assessments (% leaf area affected - disease symptoms)
> compared with unaided ones.
>
> A  sample of raters was used in each study (varied across studies). Each
> rater made two assessments (estimates of % leaf affected area) for a same
> sample of leaves (a range of values from 0 to 100%), first unaided (und)
> and then aided (aid) (pre-post test).
>
> Visual estimates from each assessment (e.g. 50 ratings) were regressed
> against “actual” values. Regression coefficients (beta1 and beta1) are
> measures of accuracy and the correlation coefficient (r) is a measure of
> precision. So, I have three variables for each rater.
>
> The data look like this:
>
> Study Rater     r_und   r_aid
> 1               1       0.65            0.76
> 1               2       0.76            0.90
> 1               3       0.80            0.90
> .               .       .               .
>
> Wolfgang has kindly helped me (more than two years ago!) to preliminary
> fit a multi-level model in metafor to summarize the gains precision
> (correlation coefficients). The effect-size was the absolute difference
> (r_aided - r_unaided) and there was a way to calculate sampling variance.
>
> This worked fine, but I have been struggling to define what would be the
> appropriate approach to analyze gains in accuracy using beta1 and beta1. In
> a primary study, vote-couting was used to infer on the value of the aid
> based on number of raters with significant (P = 0.05) departures of beta1
> and beta2 from 0 and 1, respectively, both unaided and aided.
>
> My problem is that I cannot calculate an index (such as concordance
> correlation) for accuracy because raw data was not available (I have the
> correlation coefficient, which explains in part the overall accuracy or
> concordance). So, I don’t think that an estimate of absolute difference in
> beta1 and beta2 between aided and unaided estimate for each rater make
> sense? I can see (histograms) that, in general, b1 and b2 are closer to 0
> and 1, respectively when using the aid.
>
> What I did so far was to aggregate (means) each coefficients by study to
> then obtain the sampling variance (raters as samples) for the study. Then,
> a bivariate model was fitted to these data from k studies and the estimates
> of b1 and b2 were obtained for each condition. The data looks exactly like
> shown above, but with b1_und and b1_aid, etc.
>
> I am not sure if there is a better way to analyze these data. Ideally, I
> would like to be able to calculate the concordance coefficient (which
> includes the correlation coefficient) but this seems not possible without
> the raw data.  What I have allows me to analyze separately precision and
> accuracy, but while the former seems OK (I have an effect-size and sampling
> variance for each rater), I am not sure how to better estimate gains in
> accuracy using the regression coefficients.
>
> Any thoughts or examples on this?
>
> Thanks,
>
> Emerson
>
> Prof. Emerson M. Del Ponte
> Departamento de Fitopatologia
> Universidade Federal de Viçosa
> Viçosa, MG - Brasil
> +55 (31) 3899-1103 <+55%2031%203899-1103>
> Twitter: @edelponte
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

	[[alternative HTML version deleted]]