[R] Regression with sparse matricies

[Frank E Harrell Jr]

On 05/22/2010 02:19 PM, Robin Jeffries wrote:
> I would like to run a logistic regression on some factor variables (main
> effects and eventually an interaction) that are very sparse. I have a
> moderately large dataset, ~100k observations with 1500 factor levels for one
> variable (x1) and 600 for another (X2), creating ~19000 levels for the
> interaction (X1:X2).
>
> I would like to take advantage of the sparseness in these factors to avoid
> using GLM. Actually glm is not an option given the size of the design
> matrix.
>
> I have looked through the Matrix package as well as other packages without
> much help.
>
> Is there some option, some modification of glm, some way that it will
> recognize a sparse matrix and avoid large matrix inversions?
>
> -Robin
>

Robin,

It is doubtful that fixed effects are appropriate for your situation, 
but if you do want to use them there is experimental code in the lrm 
function in the rms package to handle "strat" (strata) factors that 
makes use of the sparse matrix representation.  Not sure if it handles 
more than one factor, and you'll have to play with the code to make sure 
this method is activated.  Take a look at lrm.fit.strat.s that comes 
with the source package, the see what is needed in lrm to use it.

Frank

-- 
Frank E Harrell Jr   Professor and Chairman        School of Medicine
                      Department of Biostatistics   Vanderbilt University