[R] Reproducibility of experiment

[Marc Schwartz]

On Mon, 2007-12-17 at 16:27 +0100, darteta001 at ikasle.ehu.es wrote:
> Dear Marc and R-list,
> 
> thanks for your help. I 
> have checked Bland-Altman help 
> page about repeatability, and I learnt that instead of 
> reproducibility, 
> I was talking about repeatability. Although I am not sure whether they 
> only 
> focuse on agreement of two different measurement methods, and not 
> on 
> repeatability of one single method. 
> 
> To explain further on my topic, I have repeated ten times an 
> experiment involving protein quantification(i.e. how much protein I 
> have), 
> giving me ten continuous values. All experimental settings are similar 
> so there should be no variability due to day of experiment, operator
> or any batch effect. My aim is to know whether these ten observations 
> are good enough so that I can conclude that the repeatability of my 
> detection technique is good. But as I have learnt from Altman´s page, 
> it is not possible to set a threshold to the repeatability score to 
> say my experiment is "repeatable".  
> I guess I can obtain a 95% confidence interval for the protein 
> quantification values, 
> but I am not sure this will show how well my experiment performs. 
> Putting it differently, 
> something I would like to know is whether I can estimate
> beforehand how many times I need to run an experiment in order to be 
> confident that it is "repeatable".
> 
> 
> Thanks for your comments
> 
> David

<snip>

David,

There is information on Prof. Bland's pages pertaining to the questions
you ask. If you have not reviewed his FAQ, please do so as it covers
issues such as sample size calculations, etc.

If the 10 measures are all of the same quantity, then a simple one
sample t-test is all you need to determine whether or not the measured
values are significantly different than a presumably known correct value
and to get confidence intervals for the mean measurement.

However, if all 10 values are of the same quantity, you will not answer
the questions as to whether or not any measurement error is
constant/linear over the range of possible values and whether that error
is within "acceptable limits". This is what the Bland-Altman methods
address.

My recommendation would be to solicit local expertise in the design of
such studies, as in reality, all of this should have been specified a
priori.

In addition, both Profs. Bland and Altman participate in the MedStats
group and that would be a better forum for your queries. More
information here:

 http://groups.google.com/group/MedStats

HTH,

Marc Schwartz