[R] Ridge regression

[Ravi Varadhan]

You are right that the ans$coef and coef(ans) are different in ridge
regression, where `ans' is the object from lm.ridge.  It is the coef(ans)
that yields the coefficients on the original scale.  ans$coef is the
coefficient of "X-scaled" and "Y-centered" version.

Here is an example that illustrates the workings of ridge regression.

First let us create some data:

X1 <- runif(20)
X2 <- runif(20)
Y <- 2 * X1 - 2 * X2 + rnorm(20, sd=0.1)

lam <- 10
ans1 <- lm.ridge(Y ~ X1 + X2, lambda = lam)
ans1$coef
coef(ans1)
# Note that these two are different

# Now Let us scale the variables X1 and X2 and center Y
#
cY <- scale(Y, scale=FALSE) 
n <- length(Y)
sX1 <- scale(X1) * sqrt(n/(n-1)) 
sX2 <- scale(X2) *  sqrt(n/(n-1))

require(MASS)

lam <- 10
ans2 <- lm.ridge(cY ~ sX1 + sX2, lambda = lam)

ans2$coef
coef(ans2)
# Now, see that the coefficients of sX1 and sX2 are the same
# This is the connection!

# Armed with this insight, we now compare the ans1$coef with scaled
coefficients
#
ans1$coef
c(coef(ans1)[2] * sd(X1), coef(ans1)[3] * sd(X2)) * sqrt((n-1)/n)

# Now they are the same!

I hope this is clear.

Best,
Ravi.

----------------------------------------------------------------------------
-------

Ravi Varadhan, Ph.D.

Assistant Professor, The Center on Aging and Health

Division of Geriatric Medicine and Gerontology 

Johns Hopkins University

Ph: (410) 502-2619

Fax: (410) 614-9625

Email: rvaradhan at jhmi.edu

Webpage:
http://www.jhsph.edu/agingandhealth/People/Faculty_personal_pages/Varadhan.h
tml

 

----------------------------------------------------------------------------
--------

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On
Behalf Of David Winsemius
Sent: Wednesday, December 02, 2009 11:04 AM
To: Eleni Christodoulou
Cc: r-help at r-project.org
Subject: Re: [R] Ridge regression


On Dec 2, 2009, at 10:42 AM, Eleni Christodoulou wrote:

> Dear list,
>
> I have a couple of questions concerning ridge regression. I am using  
> the
> lm.ridge(...) function in order to fit a model to my microarray data.
> Thus *model=lm.ridge(...)*
> I retrieve some coefficients and some scales for each gene. First of  
> all, I
> would like to ask: the real coefficients of the model are not  
> included in
> the first argument of the output but in the result of coef(model),  
> am I
> right?

Not exactly. coef(model) extracts the coefficients from the model but  
the coefficients do in the example instance I created following the  
help page happen to be in the first element of the model.

eg:
 > long.rr$coef
          GNP   Unemployed Armed.Forces   Population         Year      
Employed
   25.3615288    3.3009416    0.7520553  -11.6992718   -6.5403380     
0.7864825
 > long.rr[[1]]
          GNP   Unemployed Armed.Forces   Population         Year      
Employed
   25.3615288    3.3009416    0.7520553  -11.6992718   -6.5403380     
0.7864825

> Moreover, what does the scale argument represent? Which is its
> connection with the coefficients? The R help file os not very  
> informative
> for me...

A plausible response to such a question might be that the help page is  
a sketchy substitute for the MASS book. However, I cannot find ridge  
regression in the table of contents or in the index of my copy, but I  
only have ed. 2 and the current edition is the 4th. So we will both  
need to wait for more knowledgeable (or with more recent editions of  
MASS) persons to answer that question.

(And "scales" is not an argument, rather it's a returned value.)

>
> Thank you very much in advance,
> Eleni Christodoulou
>


David Winsemius, MD
Heritage Laboratories
West Hartford, CT

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.