No. The thing to observe is, that we are dealing with continuous outcomes
(the distribution of the mean is approximately normal distributed) and if
you compare two *different* populations then their difference is *exactly*
equal to zero with probability 0.
So this is not a guideline it's a fact. The key is that when you
increase the number of samples then you increase your precision of the mean
and when the precision is high enough then you will be able to detect even
the tiniest difference (truism, given that the difference exists).
What I mean here is that if you, for instance, have two different machines
producing chewing gums and mean weight of the ones from the first is
5.00000... grams and the the mean from the ones from the second machine is
5.00012..... grams then you would with large enough samples from each
machine be able to tell that there is indeed a difference. But suppose that
the standard deviation in weight per chewing gum is 0.05 grams then the
small difference in mean weight would not influence your chewing gum
experience, since the difference between two items ,either from the same or
different machines, is so large. That's why it is relevant to think in
relevant differences. Note that I am not saying that the individual
observations are normally distributed.
Now one could say that this is frequentist mumbo jumbo since the machines
will not have a constant mean due to wear etc. But it is important when one
wishes to show that a difference is neglible e.g. when comparing two
different producers making the same pill (equivalence
trials/bioequivalence). One has to define how small neglible is to begin
with.
No statistician will say that if you compared two samples from the same
populations then their difference would be significant with
probability close to 1 if only the sample is large enough, which is what you
are trying to show when comparing dat.1 and dat.2 (their means are exactly
equal).
Best regards,
Fredrik Nilsson
2010/11/27 Daniel Ezra Johnson
> On 11/24/10 07:59, Rolf Turner wrote:
> > >>
> > >> It is well known amongst statisticians that having a large enough data
> set will
> > >> result in the rejection of *any* null hypothesis, i.e. will result in
> a small
> > >> p-value.
>
> This seems to be a well-accepted guideline, probably because in the
> social sciences, usually, none of the predictors truly has an effect
> size of zero.
> However, unless I am misunderstanding it, the statement appears to me
> to be more generally false.
> For example, when the population difference of means actually equals
> zero, in a t-test, very large sample sizes do not lead to small
> p-values.
>
> set.seed(1)
> n <- 1000000 # 10^6
> dat.1 <- rnorm(n/2,0,1)
> dat.2 <- rnorm(n/2,0,1)
> t.test(dat.1,dat.2,var.equal=T)
> # p = 0.60
>
> set.seed(1)
> n <- 10000000 # 10^7
> dat.1 <- rnorm(n/2,0,1)
> dat.2 <- rnorm(n/2,0,1)
> t.test(dat.1,dat.2,var.equal=T)
> # p = 0.48
>
> set.seed(1)
> n <- 100000000 # 10^8
> dat.1 <- rnorm(n/2,0,1)
> dat.2 <- rnorm(n/2,0,1)
> t.test(dat.1,dat.2,var.equal=T)
> # p = 0.80
>
> Such results - where the null hypothesis is NOT rejected - would
> presumably also occur in any experimental situations where the null
> hypothesis was literally true, regardless of the size of the data set.
> No?
>
> Daniel
>
> _______________________________________________
> R-sig-mixed-models@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
[[alternative HTML version deleted]]