Hi Peter, searched old mail archive and found this topic had been discussed before. The previous discussion was around a situation where there was a very large sample size involved so even a small effect still showed up as significant even with low R square of the model.

In my case, the sample size is 72, the significance of "group" effect is due to large effect relative to its standard error:

obj<-lm(y~age+sex+school+group,dat)

summary(obj)

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)   169.8634    13.4678  12.613   <2e-16
age            -0.3737     0.2762  -1.353   0.1805
sexM            2.1137     8.6585   0.244   0.8079
schoolS2        4.1711     8.1811   0.510   0.6118
groupG2      -20.8944    10.2807  -2.032   0.0461

Residual standard error: 32.13 on 67 degrees of freedom
Multiple R-squared: 0.1732,     Adjusted R-squared: 0.1238 
F-statistic: 3.509 on 4 and 67 DF,  p-value: 0.01163 

So R-squared is quite low (0.17), what's your opinion on the argument that the significant coefficient for group is not trustworthy because the model variance was not sufficiently accounted for, and if additional factors could be identified and included in the model, that might changed the effect of group from significant to insignificant.

Many thanks for sharing your thoughts.

John




________________________________

To: peter dalgaard <pdalgd@gmail.com> 
Cc: "r-help@r-project.org" <r-help@r-project.org> 
Sent: Tuesday, May 8, 2012 1:45 PM
Subject: Re: [R] low R square value from ANCOVA model


Thanks again Peter. What about the argument that because low R square (e.g. R^2=0.2) indicated the model variance was not sufficiently explained by the factors in the model, there might be additional factors that should be identified and included in the model. And If these additional factors were indeed included, it might change the significance for the factor of interest that previously showed significant coefficient. In other word, if R square is low, the significant coefficient observed is not trustworthy.

What's your opinion on this argument?

Many thanks!

John



________________________________
 From: peter dalgaard <pdalgd@gmail.com>

Cc: "r-help@r-project.org" <r-help@r-project.org> 
Sent: Monday, May 7, 2012 11:43 PM
Subject: Re: [R] low R square value from ANCOVA model


On May 8, 2012, at 08:34 , array chip wrote:

> Thank you Peter, so if I observe a significant coefficient, that significance still holds because the standard error of the coefficient has taken the residual error (which is large because large R square) into account, am I correct?

In essence, yes. One might quibble over the use of "large because", but it's not important for the main point.

-pd

> John
> From: peter dalgaard <pdalgd@gmail.com>

> Cc: "r-help@r-project.org" <r-help@r-project.org> 
> Sent: Monday, May 7, 2012
 11:07 PM
> Subject: Re: [R] low R square value from ANCOVA model
> 
> 
> On May 8, 2012, at 05:10 , array chip wrote:
> 
> > Hi, what does a low R-square value from an ANCOVA model mean? For example, if the R square from the model is about 0.2, does this mean the results should NOT be trusted? I checked the residuals of the model, it looked fine...
> 
> It just means that your model has low predictive power (at the individual level). I.e. the noise (error) part of the model is large relative to the signal (systematic part). Statistical inferences are not compromised by that, except of course that large error variation is reflected in large standard errors of estimated regression coefficients. 
> 
> >  
> > Thanks for any suggestion.
> >  
> > John
> >     [[alternative HTML version deleted]]
> > 
> >
 ______________________________________________
> > R-help@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> 
> -- 
> Peter Dalgaard, Professor,
> Center for Statistics, Copenhagen Business School
> Solbjerg Plads 3, 2000 Frederiksberg, Denmark
> Phone: (+45)38153501
> Email: pd.mes@cbs.dk  Priv: PDalgd@gmail.com
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
Peter Dalgaard, Professor
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Email: pd.mes@cbs.dk  Priv: PDalgd@gmail.com
	[[alternative HTML version deleted]]

