[R] Stochastic Gradient Ascent for logistic regression

[Tim LIU]

Hi. guys,

I am trying to write my own Stochastic Gradient Ascent for logistic 
regression in R. But it seems that I am having convergence problem.

Am I doing anything wrong, or just the data is off?

Here is my code in R -



lbw <-
read.table("http://www.biostat.jhsph.edu/~ririzarr/Teaching/754/lbw.dat"
, header=TRUE)

attach(lbw)



lbw[1:2,]
low age lwt race smoke ptl ht ui ftv bwt
1 0 19 182 2 0 0 0 1 0 2523
2 0 33 155 3 0 0 0 0 3 2551




#-----R implementation of logistic regression : gradient descent ------
sigmoid<-function(z)
{
1/(1 + exp(-1*z))

}




X<-cbind(age,lwt, smoke, ht, ui)

#y<-low


my_logistic<-function(X,y)
{

alpha <- 0.005
n<-5 
m<-189
max_iters <- 189 #number of obs

ll<-0

X<-cbind(1,X)

theta <-rep(0,6) # intercept and 5 regerssors
#theta <- c(1.39, -0.034, -0.01, 0.64, 1.89, 0.88) #glm estimates as 
starting values
theta_all<-theta
for (i in 1:max_iters) 
{ 
dim(X)
length(theta)
hx <- sigmoid(X %*% theta) # matrix 
product

ix<-i

for (j in 1:6)
{
theta[j] <- theta[j] + alpha * ((y-hx)[ix]) * X[ix,j] 
#stochastic gradient !

}





logl <- sum( y * log(hx) + (1 - y) * log(1 - hx) ) #direct 
multiplication

ll<-rbind(ll, logl)


theta_all = cbind(theta_all,theta)
}

par(mfrow=c(4,2))


plot(na.omit(ll[,1]))
lines(ll[,1])

for (j in 1:6)
{

plot(theta_all[j,])
lines(theta_all[j,])
} 


#theta_all
#ll
cbind(ll,t(theta_all))
}


my_logistic(X,low)
==============


parameter estimates values jumped after 130+ iterations...

not converging even when I use parameter estimates as starting values 
from glm (family=binomial)


help!




-- 
View this message in context: http://www.nabble.com/Stochastic-Gradient-Ascent-for-logistic-regression-tp23239378p23239378.html
Sent from the R help mailing list archive at Nabble.com.