[R] correlation significance testing with multiple factor levels

[Greg Snow]

If I understand your problem correctly, then one option is to use a permutation test.  This tests the null hypothesis that set A and set B are identical and the only reason you are seeing a difference is due to random chance.  The procedure is to randomly shuffle the observations between the sets and recomputed your test statistic (difference of mean of correlations).

Here is one example of how you could do this using some simulated data (you should probably try different rho matrices to see how this performs under different situations).

library(MASS)

rho1 <- matrix(.6, 3, 3)
diag(rho1) <- 1
rho2 <- matrix(.1, 3, 3)
diag(rho2) <- 1

x1 <- mvrnorm(25, rep(0,3), rho1)
x2 <- mvrnorm(25, rep(0,3), rho2)

x <- rbind(x1,x2)
g <- rep(1:2, each=25)

tsfunc <- function(g){
	c1 <- cor(x[g==1,])
	c2 <- cor(x[g==2,])
	m1 <- mean( c1[ lower.tri(c1) ] )
	m2 <- mean( c2[ lower.tri(c2) ] )
	m1-m2
}

out <- replicate(999, tsfunc( sample(g) ) )
out <- c(tsfunc(g), out)

hist(out)
abline(v=out[1])

mean( out >= out[1] )  # one-sided p-value


hope this helps,

-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at imail.org
801.408.8111


> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-
> project.org] On Behalf Of Jack Tanner
> Sent: Thursday, January 21, 2010 4:08 PM
> To: r-help at stat.math.ethz.ch
> Subject: [R] correlation significance testing with multiple factor
> levels
> 
> [Apologies in advance if this is too "statistics" and not enough "R".]
> 
> I've got an experiment with two sets of treatments. Each subject either
> received
> all treatments from set A or all treatments from set B.
> 
> I can compute the N pairwise correlations for all treatments in either
> set using
> cor(). If I take the mean of these N pairwise correlations, I see that
> the
> effects of treatments in set A are much more correlated than the
> effects of
> treatments in set B. (Mean correlation for set A is 0.6, mean for set B
> is 0.1).
> This is probably wrongheaded, but I'd like to be able to report whether
> this is
> a significant difference. I know about cor.test(), but I don't know
> whether/how
> I can adapt that for my use case.
> 
> Thanks in advance for your advice.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.