What I tell my students in my experimental design class is that 
experiments need to have both internal validity and external validity.

Internal validity means that we can reproduce our results by a 
similar experiment.  This implies that we need to control extraneous 
factors that might affect the outcome of interest.

External validity means that we can make inference to the population 
of interest.  This implies that we need to assess the variability of 
the response in the population at large, which is in opposition to 
controlling variation to ensure internal validity.

The strength of the randomized complete block design (i.e. batches 
grown separately and all conditions measured on each block) is that 
we can use tight control within block and do comparisons that are 
internally valid, while allowing our blocks to reflect the population 
variability.

We should not pool samples across different blocks.  We need an 
assessment of the between block variability to assess the 
reproducibility of the results in the entire population.  There is a 
nice literature on pooling (e.g. Kendziorski and colleagues which 
includes experimental results or my paper in Applied Bioinformatics, 
2004 which gives theoretical results based on a 2nd order mixed model).

--Naomi

At 09:20 AM 1/20/2006, Suresh Gopalan wrote:
>I agree that independent replication is the best bet as of now, 
>though it has the risk of introduction of new defects/hidden 
>variables that influence the phenotype in question, which may indeed 
>be relevant or constitute another line of study.
>
>If one decides to take this risk or for other reasons does 
>experiments in perfectly identical (reproducible?) conditions and 
>take replicates pooled from a very large population (50 plants each 
>in 3 replicates) as mentioned below: if that removes some 
>variability inherent to each plant, so be it. Isn't the goal to 
>study the variable of interest masking the irrelevant variables (at 
>least in that study)?  How would this make statistical testing invalid?
>
>I wonder if in either case it is any different or worse than the 
>normalization schemes and assumptions used in many of the currently 
>used popular analysis or summary schemes?
>
>Suresh
>
>(Suresh Gopalan, Ph.D)
>
>
>----- Original Message ----- From: "Matthew Hannah" <Hannah@mpimp-golm.mpg.de>
>To: "Naomi Altman" <naomi@stat.psu.edu>; <fhong@salk.edu>
>Cc: <bioconductor@stat.math.ethz.ch>
>Sent: Friday, January 20, 2006 6:02 AM
>Subject: Re: [BioC] Biological replication (was RNA degradation problem)
>
>
>>>
>>>The question of what is appropriate biological replication is
>>>a tough one.  The objective is to obtain results that are
>>>valid in the population of interest, which usually is not
>>>plants grown in a single batch in the green house.  But how
>>>much variability should we induce?  Each batch of plants
>>>grown separately but in the same building (different growth
>>>chambers), grown in different labs?  different universities?
>>
>>Yes, but this is more a question of 'some' biological replication versus
>>none. Obviously, if you have perfect reproducability of your growth
>>conditions then repeat experiments will have little influence, but in my
>>experience independent experiments suitably accounts for slight
>>environmental and sampling (eg:time) variability. Plants grown under the
>>same conditions are highly reproducable, so even the random block design
>>might not be ideal depending on what the environmental factors are -
>>light, water, temp etc.. I would always favour separate, independent
>>experiments.
>>
>>As for reproducability in general this is a problem. I'm sure in all
>>fields that some patterns found by a certain lab, labelling, scanning
>>etc.. will not be reproducible. For example, I wonder how many training
>>set - sample set molecular diagnosis studies would continue to work if
>>new independent data is introduced without updating the whole study.
>>
>>>In my very first Affy experiment, the investigator did the
>>>following:  2 batches of plants grown separately, 2 samples
>>>of plants from 1 of the batches, 2 microarrays from one of
>>>the samples, for 4 arrays in all.
>>>The correlation among the results was 2 arrays from same
>>>sample > 2 samples from same batch > 2 batches.  This should
>>>be no surprise, even though we did not have enough
>>>replication to do any formal testing.
>>>
>>>I think at minimum that you want to achieve results that
>>>would be replicable within your own lab.  That would suggest
>>>batches of plants grown separately from separate batches of seed.
>>>
>>>The best plan is a randomized complete block design, with
>>>every condition sampled in every block.  If the conditions
>>>are tissues, this is readily achieved.
>>
>>I assume you mean random in each independent experiment, and then
>>independently repeated, in which case this is the best approach.
>>
>>>Personally, I look at the density plots of the probes on the
>>>arrays.  If they have the same "shape" (which is usually a
>>>unimodal distribution with long tail to the right on the log2
>>>scale) then I cross my fingers (that is supposed to bring
>>>good luck) and use RMA.  Most of the experiments I have been
>>>involved with using arabidopsis arrays have involved tissue
>>>differences, and the amount of differential expression has
>>>been huge on the probeset scale (over 60% of genes), but
>>>these probe densities have been pretty similar.
>>
>>I always look at RNAdeg and PLM as well, but in most cases this is also
>>seen on the density plots.
>>
>>Cheers,
>>MAtt
>>
>>> >From: fhong@salk.edu [mailto:fhong@salk.edu]
>>> >Sent: Thu 19/01/2006 21:27
>>> >To: Matthew Hannah
>>> >Cc: bioconductor@stat.math.ethz.ch
>>> >Subject: Re: [BioC] RNA degradation problem
>>> >
>>> >
>>> >
>>> >Hi Matthew,
>>> >
>>> >Thank you very much for your help.
>>> >
>>> > > >It's amazing how many
>>> > >> lab plant biologists see pooled samples from a bulk of
>>>plants grown
>>> > >> at the same time as biological replicates when they are
>>>clearly not.
>>> > >I would think that all plants under experiment shoudl be
>>>grown at the
>>> > >same time without different conditions/treatments. Biological
>>> > >replicates should be tissue samples from differnt groupd
>>>of plants,
>>> > >say sample from 50 plants as replicate1 and sample from
>>>another 50 as replicate 2.
>>> > >Do you think that biological replicates should be grown at
>>>different time?
>>> >
>>> >
>>> >Absolutely! Biological replication must be either single
>>>plants grown
>>> >in the same experiment (but noone wants to risk single plants for
>>> >arrays) or large pools of plants from INDEPENDENT
>>>experiments (or the
>>> >pools must be smaller than sample size - doesn't really happen for
>>> >arrays) otherwise what biological variability are you
>>>sampling? Say you
>>> >have 150 plants growing in the greenhouse and you harvest 3 random
>>> >pools of 50 as your 3 'biological replicates' then you will have
>>> >eliminated all variability from them and the arrays will be
>>>as good as
>>> >technical replicates and any statistical testing is invalid.
>>> >
>>> > >> I find hist, RNA deg, AffyPLM and a simple RMA norm followed by
>>> > >> plot(as.data.frame(exprs(eset.rma))) can answer in most
>>>cases for
>>> > >> why it didn't work, or won't work - in the rare case
>>>when someone
>>> > >> asks for QC
>>> > > >before rather than after they realise the data is strange ;-)
>>> > >This actually pull out another question: when % of
>>>differential genes
>>> > >is large, which normalization better works better?
>>> >I've posted on this alot about 1.5 years ago, you should
>>>find it in the
>>> >archives - but simply noone knows or has tested it
>>> >
>>> >
>>> > >http://cactus.salk.edu/temp/QC_t.doc
>>> > >Take a look at the last plot, which clearly indicate homogeneous
>>> > >within replicates and heterogeneous among samples.
>>> > >(1) Will stem top and stem base differ so much? Or it is the
>>> > >preparation process bring in extra correlaton within replicates.
>>> > >(2) when % of differential genes is large, which
>>>normalization better
>>> > >works better?
>>> >Looking at these scatterplots, I can honestly say I've never seen so
>>> >much DE. I would be suprised if samples such as different stem
>>> >positions were so different. Something must be wrong with
>>>the samples
>>> >or sampling in my opinion. The scatterplots are slightly more user
>>> >friendly if you use pch="."
>>> >
>>> >HTH,
>>> >
>>> >Matt
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >--------------------
>>> >Fangxin Hong  Ph.D.
>>> >Plant Biology Laboratory
>>> >The Salk Institute
>>> >10010 N. Torrey Pines Rd.
>>> >La Jolla, CA 92037
>>> >E-mail: fhong@salk.edu
>>> >(Phone): 858-453-4100 ext 1105
>>> >
>>> >_______________________________________________
>>> >Bioconductor mailing list
>>> >Bioconductor@stat.math.ethz.ch
>>> >https://stat.ethz.ch/mailman/listinfo/bioconductor
>>>
>>>Naomi S. Altman                                814-865-3791 (voice)
>>>Associate Professor
>>>Dept. of Statistics                              814-863-7114 (fax)
>>>Penn State University                         814-865-1348
>>>(Statistics)
>>>University Park, PA 16802-2111
>>>
>>
>>_______________________________________________
>>Bioconductor mailing list
>>Bioconductor@stat.math.ethz.ch
>>https://stat.ethz.ch/mailman/listinfo/bioconductor
>
>

Naomi S. Altman                                814-865-3791 (voice)
Associate Professor
Dept. of Statistics                              814-863-7114 (fax)
Penn State University                         814-865-1348 (Statistics)
University Park, PA 16802-2111

	[[alternative HTML version deleted]]

