[R] bootstrapping in regression

[Greg Snow]

What you are describing is actually a permutation test rather than a bootstrap (related concepts but with a subtle but important difference).

The way to do a permutation test with multiple x's is to fit the reduced model (use all x's other than x1 if you want to test x1) on the original data and store the fitted values and the residuals.

Permute the residuals (randomize their order) and add them back to the fitted values and fit the full model (including x1 this time) to the permuted data set.  Do this a bunch of times and it will give you the sampling distribution for the slope on x1 (or whatever your set of interest is) when the null hypothesis that it is 0 given the other variables in the model is true.

Permuting just x1 only works if x1 is orthogonal to all the other predictors, otherwise the permuting destroys the relationship with the other predictors and does not do the test you want.

Bootstrapping depends on sampling with replacement, not permuting, and is used more for confidence intervals than for tests (the reference by John Fox given to you in another reply can help if that is the approach you want to take).

Hope this helps,

-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at imail.org
801.408.8111


> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-
> project.org] On Behalf Of Thomas Mang
> Sent: Thursday, January 29, 2009 9:44 AM
> To: r-help at stat.math.ethz.ch
> Subject: [R] bootstrapping in regression
> 
> Hi,
> 
> Please apologize if my questions sounds somewhat 'stupid' to the
> trained
> and experienced statisticians of you. Also I am not sure if I used all
> terms correctly, if not then corrections are welcome.
> 
> I have asked myself the following question regarding bootstrapping in
> regression:
> Say for whatever reason one does not want to take the p-values for
> regression coefficients from the established test statistics
> distributions (t-distr for individual coefficients, F-values for
> whole-model-comparisons), but instead apply a more robust approach by
> bootstrapping.
> 
> In the simple linear regression case, one possibility is to randomly
> rearrange the X/Y data pairs, estimate the model and take the
> beta1-coefficient. Do this many many times, and so derive the null
> distribution for beta1. Finally compare beta1 for the observed data
> against this null-distribution.
> 
> What I now wonder is how the situation looks like in the multiple
> regression case. Assume there are two predictors, X1 and X2. Is it then
> possible to do the same, but just only rearranging the values of one
> predictor (the one of interest) at a time? Say I want again to test
> beta1. Is it then valid to many times randomly rearrange the X1 data
> (and keeping Y and X2 as observed), fit the model, take the beta1
> coefficient, and finally compare the beta1 of the observed data against
> the distributions of these beta1s ?
> For X2, do the same, randomly rearrange X2 all the time while keeping Y
> and X1 as observed etc.
> Is this valid ?
> 
> Second, if this is valid for the 'normal', fixed-effects only
> regression, is it also valid to derive null distributions for the
> regression coefficients of the fixed effects in a mixed model this way?
> Or does the quite different parameters estimation calculation forbid
> this approach (Forbid in the sense of bogus outcome) ?
> 
> Thanks, Thomas
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.