[R-meta] MA of regression coefficients (as accuracy indicators) in pre-post test

[Emerson M. Del Ponte]

Dear All,

I have collected data from primary studies where an assessment aid (diagrams) was developed and tested for improvements in accuracy and precision of visual assessments (% leaf area affected - disease symptoms) compared with unaided ones.

A  sample of raters was used in each study (varied across studies). Each rater made two assessments (estimates of % leaf affected area) for a same sample of leaves (a range of values from 0 to 100%), first unaided (und) and then aided (aid) (pre-post test). 

Visual estimates from each assessment (e.g. 50 ratings) were regressed against “actual” values. Regression coefficients (beta1 and beta1) are measures of accuracy and the correlation coefficient (r) is a measure of precision. So, I have three variables for each rater.

The data look like this:

Study Rater	r_und	r_aid
1		1	0.65		0.76
1		2	0.76		0.90
1		3 	0.80		0.90
.		.	.		.

Wolfgang has kindly helped me (more than two years ago!) to preliminary fit a multi-level model in metafor to summarize the gains precision (correlation coefficients). The effect-size was the absolute difference (r_aided - r_unaided) and there was a way to calculate sampling variance. 

This worked fine, but I have been struggling to define what would be the appropriate approach to analyze gains in accuracy using beta1 and beta1. In a primary study, vote-couting was used to infer on the value of the aid based on number of raters with significant (P = 0.05) departures of beta1 and beta2 from 0 and 1, respectively, both unaided and aided. 

My problem is that I cannot calculate an index (such as concordance correlation) for accuracy because raw data was not available (I have the correlation coefficient, which explains in part the overall accuracy or concordance). So, I don’t think that an estimate of absolute difference in beta1 and beta2 between aided and unaided estimate for each rater make sense? I can see (histograms) that, in general, b1 and b2 are closer to 0 and 1, respectively when using the aid. 

What I did so far was to aggregate (means) each coefficients by study to then obtain the sampling variance (raters as samples) for the study. Then, a bivariate model was fitted to these data from k studies and the estimates of b1 and b2 were obtained for each condition. The data looks exactly like shown above, but with b1_und and b1_aid, etc. 

I am not sure if there is a better way to analyze these data. Ideally, I would like to be able to calculate the concordance coefficient (which includes the correlation coefficient) but this seems not possible without the raw data.  What I have allows me to analyze separately precision and accuracy, but while the former seems OK (I have an effect-size and sampling variance for each rater), I am not sure how to better estimate gains in accuracy using the regression coefficients.

Any thoughts or examples on this?

Thanks,

Emerson

Prof. Emerson M. Del Ponte
Departamento de Fitopatologia
Universidade Federal de Viçosa
Viçosa, MG - Brasil
+55 (31) 3899-1103
Twitter: @edelponte