> sessionInfo()
R version 2.13.1 (2011-07-08)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)

locale:
[1] en_US.UTF-8/en_US.UTF-8/C/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Matrix_0.999375-50 lattice_0.19-30   

loaded via a namespace (and not attached):
[1] grid_2.13.1  tools_2.13.1

Dear R-help,

I am using the Matrix (0.999375-50) and MatrixModels (0.2-1) packages to estimate regressions with 14,791 fixed effects. I use the function sparse.model.matrix() to partial out the fixed effects and then use lm() and an adjustment to the standard errors to study the four regressors that I am interested in.

I want to calculate standardized residuals and need the diagonal elements of the hat matrix. I tried to adjust function hat() in the "stats" package for use with sparse matrices. But the method qr.qy does not seem to have a function available for the following signature: signature(qr = "sparseQR", y = "dgCMatrix") - although it indicates that it understands this signature.

Following is a minimal example (in my application dd2 is too large for my memory):

library(Matrix)


data(KNex); 
mm <- KNex$mm
mmQR <- qr(mm)
dim(mmQR) # 1850  712



class(mmQR) # [1] "sparseQR";attr(,"package");"Matrix"

dd1 <- sparseMatrix(1:ncol(mm),1:ncol(mm), x = 1, dims = c(nrow(mm),ncol(mm)))

class(dd1) # dgCMatrix;attr(,"package");Matrix

dd2 <- diag(1,ncol = ncol(mm) , nrow = nrow(mm))

class(dd2) #matrix

showMethods("qr.qy")
#Function: qr.qy (package base)
#qr="ANY", y="ANY"
#qr="sparseQR", y="dgCMatrix"
#    (inherited from: qr="ANY", y="ANY")
#qr="sparseQR", y="dgeMatrix"
#qr="sparseQR", y="matrix"
#qr="sparseQR", y="numeric"


dhat1 <- rowSums(qr.qy(qr = mmQR, y = dd1)^2) # Error in rowSums(qr.qy(qr = mmQR, y = dd1)^2) : 
  # error in evaluating the argument 'x' in selecting a method for function 'rowSums': 
  # Error in qr.qy(qr = mmQR, y = dd1) : argument is not a QR decomposition


dhat2 <- rowSums(qr.qy(qr = mmQR, y = dd2)^2) # Works fine


I would be grateful for pointers how to add a function for Matrix:::qr.qy and signature(qr = "sparseQR", y = "dgCMatrix"). What low level functions do I have to call?
I would also be grateful for any suggestions for finding diagonal values of the hat matrix in regression problems with large and sparse design matrices.


Best,

Felix Lamp
(Erasmus University Rotterdam, School of Economics)





	[[alternative HTML version deleted]]

