[R] Bug : Autocorrelation in sample drawn from stats::rnorm (hmh)

[William Bell]

Hi Hugo,
I've been able to replicate your bug, including for other distributions (runif, rexp, rgamma, etc) which shouldn't be surprising since they're probably all drawing from the same pseudo-random number generator.  Interestingly, it does not seem to depend on the choice of seed, I am not sure why that is the case.
I'll point out first of all that the R-devel mailing list is perhaps better suited for this query, I'm fairly sure we're supposed to direct bug reports, etc there.
It is possible this is a known quantity but is tolerated, I could think of many reasons why that might be the case, not least of which being that as far as I know, the vast majority of Monte Carlo methods involve >>40 trials (which seems to be enough for the effect to disappear), with the possible exception of procedures for testing the power of statistical tests on small samples?
There might be more to be said, but I thought I'd just add what I could from playing around with it a little bit.
For anyone who wishes to give it a try, I suggest this implementation of the autocorrelation tester which is about 80 times faster:
DistributionAutocorrelation_new <- function(SampleSize)    {
    Cor <- replicate(1e5, function() {X <- rnorm(SampleSize)    return(cor(X[-1], X[-length(X)]))})    return(Cor)}
I have the same Stats package version installed.
- (Thomas) William BellHons BSc Candidate (Biology and Mathematics)BA Candidate (Philosophy)McMaster University
# Hi,# # # I just noticed the following bug:#   #   When we draw a random sample using the function stats::rnorm, there # should be not auto-correlation in the sample. But their is some # auto-correlation _when the sample that is drawn is small_.# # I describe the problem using two functions:#   #   DistributionAutocorrelation_Unexpected which as the wrong behavior : #   _when drawing some small samples using rnorm, there is generally a # strong negative auto-correlation in the sample_.# # and# # DistributionAutocorrelation_Expected which illustrate the expected behavior# # # # *Unexpected : *#   #   DistributionAutocorrelation_Unexpected = function(SampleSize){#     Cor = NULL#     for(repetition in 1:1e5){#       X = rnorm(SampleSize)#       Cor[repetition] = cor(X[-1],X[-length(X)])#     }#     return(Cor)#   }# # par(mfrow=c(3,3))# for(SampleSize_ in c(4,5,6,7,8,10,15,20,50)){#   hist(DistributionAutocorrelation_Unexpected(SampleSize_),col='grey',main=paste0('SampleSize=',SampleSize_)) #   ; abline(v=0,col=2)# }# # output:#   #   #   *Expected**:*#   #   DistributionAutocorrelation_Expected = function(SampleSize){#     Cor = NULL#     for(repetition in 1:1e5){#       X = rnorm(SampleSize)#       *    Cor[repetition] = cor(sample(X[-1]),sample(X[-length(X)]))*#     }#     return(Cor)#   }# # par(mfrow=c(3,3))# for(SampleSize_ in c(4,5,6,7,8,10,15,20,50)){#   hist(DistributionAutocorrelation_Expected(SampleSize_),col='grey',main=paste0('SampleSize=',SampleSize_)) #   ; abline(v=0,col=2)# }# # # # # Some more information you might need:#   #   #   packageDescription("stats")# Package: stats# Version: 3.5.1# Priority: base# Title: The R Stats Package# Author: R Core Team and contributors worldwide# Maintainer: R Core Team <R-core using r-project.org>#   Description: R statistical functions.# License: Part of R 3.5.1# Imports: utils, grDevices, graphics# Suggests: MASS, Matrix, SuppDists, methods, stats4# NeedsCompilation: yes# Built: R 3.5.1; x86_64-pc-linux-gnu; 2018-07-03 02:12:37 UTC; unix# # Thanks for correcting that.# # fill free to ask any further information you would need.# # cheers,# # hugo# # # -- #   - no title specified# # Hugo Mathé-Hubert# # ATER# # Laboratoire Interdisciplinaire des Environnements Continentaux (LIEC)# # UMR 7360 CNRS -  Bât IBISE# # Université de Lorraine  -  UFR SciFA# # 8, Rue du Général Delestraint# # F-57070 METZ# # +33(0)9 77 21 66 66# - - - - - - - - - - - - - - - - - -#   Les réflexions naissent dans les doutes et meurent dans les certitudes. # Les doutes sont donc un signe de force et les certitudes un signe de # faiblesse. La plupart des gens sont pourtant certains du contraire.# - - - - - - - - - - - - - - - - - -#   Thoughts appear from doubts and die in convictions. Therefore, doubts # are an indication of strength and convictions an indication of weakness. # Yet, most people believe the opposite.
	[[alternative HTML version deleted]]