[R-sig-ME] Mixed-models and condition number

[Stephan Kolassa]

Hi Christina,

if centering and/or scaling the columns of design matrices reduces 
kappa, that's nice and dandy - but of course the question remains what 
this means for the stability of your linear system, which is what you 
are really interested in.

An aside: kappa depends on the matrix norm you are using, and all 
sensitivity and other results will then be expressed in a compatible 
vector norm. The 2-norm is the most widely used one (and kappa_2 can be 
easily calculated as the ratio of the largest to the smallest 
eigenvalue), but both the 1-norm and the \infty-norm may be more useful, 
depending on your application and interest.

Getting a mathematician interested in your problem sounds like a great 
idea! Looking at uni-bayreuth.de, I guess the engineering mathematicians 
are your best bet:
http://www.ingenieurmathematik.uni-bayreuth.de/forschung.html
Mathe V seems to be more focused on dynamical systems, this may not be 
their core interest.

Good luck!
Stephan


Christina Bogner schrieb:
> Dear Stephan,
> 
> thank your very much for your response and the detailed list of 
> literature. I knew about Belsley (1991b) and used it on the design 
> matrix of the fixed-effects. My (absolutely empirical) results were the 
> following:
> 
> on  a small data set of 58 values and the mixed-effects model:
> mymodel=lme(log(calcium) ~ soil.horizon+flow.region+content.of.silt, 
> data=mydata, random=~1|plot)
> with soil.horizon and flow.region: factors
> content.of.silt: continuous covariate
> 
> 1. mean-centering the continuous covariate decreased the collinearity 
> between the intercept term and the continuous covariate 
> (summary.lme$corFixed and Belsley 1991b on the design matrix of 
> fixed-effects) and decreased kappa (of the design matrix of 
> fixed-effects) by factor 12.
> 2. scaling the covariate to obtain fixed-effects estimates of comparable 
> size decreased kappa by factor 4, but had no effect on correlation of 
> the fixed-effects.
> 3. I compared kappas of the mixed-effects design matrix and as proposed 
> by Douglas Bates of the "triangular matrix derived from the 
> fixed-effects model matrix after removing the random effects" in lme4: 
> the influence of mean-centering and scaling on kappa was comparable and 
> values of kappas of the triangular matrix and the design matrix of the 
> fixed-effects differed little for mean-centered and scaled model, but 
> largely for the non-scaled and non-centered one.
> 
> I will try to find a mathematician at my university who would like to 
> play around with mixed-models  ;-).
> 
> Thanks again
> 
> Christina
> 
> 
> Stephan Kolassa schrieb:
>> Hi Christina,
>> let me start by saying that I don't know of anyone looking at
>> conditioning of design matrices in a mixed model environment. Might be a
>> nice topic to have an M. Sc. student play around with empirically. The 
>> problem with ill-conditioning in fixed-effects models basically comes 
>> down to high variances in the parameter estimates, so one could 
>> actually build a mixed model with an ill-conditioned design matrix and 
>> play around with small changes to simulated observations, checking 
>> whether inferences or estimates exhibit "large" variance.
>>
>> If you find out anything about this, would you let me know?
>>
>> That said, my recent interest has been in collinearity between
>> predictors, which is not exactly conditioning, but reasonably close to
>> it. I'd recommend you look at Hill & Adkins (2001) and the collinearity
>> diagnostics they recommend. Belsley (1991a) wrote an entire monograph
>> about them, but there are also shorter introductions, e.g., Belsley 
>> (1991b).
>>
>> Scaling the columns of X to equal euclidean length (usually to length 1)
>> before diagnosing collinearity appears to be accepted procedure, so I 
>> think scaling would be a good starting point in the mixed model, too. 
>> However, there is a discussion as to whether to first remove the
>> constant column from X and subtract the column mean from each of the 
>> remaining columns.
>>
>> Marquardt (1980) claims that centering removes "nonessential ill
>> conditioning." Weisberg (1980) and Montgomery and Peck (1982) also 
>> advocate centering.
>>
>> Other practitioners maintain that centering removes meaningful
>> information from X, such as collinearity with the constant column, and 
>> should not be used (Belsey et al., 1980; Belsley, 1984a, 1984b, 1986, 
>> 1991a, 1991b; Echambadi & Hess, 2007; Hill & Adkins, 2001). For 
>> example, Simon and Lesage (1988) found that collinearity with the 
>> constant
>> column introduces numerical instability, which is mitigated but not 
>> prevented by employing collinearity diagnostics after centering X. In 
>> addition, these problems are not confined to the constant coefficient, 
>> but extend to all estimates.
>>
>> For a very lively debate on this topic see Belsley (1984a); Cook (1984);
>> Gunst (1984); Snee and Marquardt (1984); Wood (1984); Belsley (1984b). 
>> The consensus seems to be that centering cannot be once and for all be 
>> advised or rejected; rather, whether or not to center data depends on 
>> the problem one is facing.
>>
>> HTH,
>> Stephan
>>
>>
>> * Belsey, D. A., Kuh, E., & Welsch, R. E. (1980). Regression 
>> Diagnostics: Identifying Influential Data and Sources of Collinearity. 
>> New York, NY: John Wiley & Sons.
>>
>> * Belsley, D. A. (1984a, May). Demeaning Conditioning Diagnostics 
>> through Centering. The American Statistician, 38(2), 73-77.
>>
>> * Belsley, D. A. (1984b, May). Demeaning Conditioning Diagnostics 
>> through Centering: Reply. The American Statistician, 38(2), 90-93.
>>
>> * Belsley, D. A. (1986). Centering, the constant, first-differencing, 
>> and assessing conditioning. In E. Kuh & D. A. Belsley (Eds.), Model 
>> Reliability (p. 117-153). Cambridge: MIT Press.
>>
>> * Belsley, D. A. (1987). Collinearity and Least Squares Regression: 
>> Comment -- Well-Conditioned Collinearity Indices. Statistical Science, 
>> 2(1), 86-91. Available from http://projecteuclid.org/euclid.ss/1177013441
>>
>> * Belsley, D. A. (1991a). Conditioning Diagnostics: Collinearity and 
>> Weak Data in Regression. New York, NY: Wiley.
>>
>> * Belsley, D. A. (1991b, February). A Guide to using the collinearity 
>> diagnostics. Computational Economics, 4(1), 33-50. Available from 
>> http://www.springerlink.com/content/v135h6631x412kk8/
>>
>> * Cook, R. D. (1984, May). Demeaning Conditioning Diagnostics through 
>> Centering: Comment. The American Statistician, 38(2), 78-79.
>>
>> * Echambadi, R., & Hess, J. D. (2007, May-June). Mean-Centering Does 
>> Not Alleviate Collinearity Problems in Moderated Multiple Regression 
>> Models. Marketing Science, 26(3), 438-445.
>>
>> * Golub, G. H., & Van Loan, C. F. (1996). Matrix Computations (3rd 
>> ed.). Baltimore: Johns Hopkins University Press.
>>
>> * Gunst, R. F. (1984, May). Comment: Toward a Balanced Assessment of 
>> Collinearity Diagnostics. The American Statistician, 38(2), 79-82.
>>
>> * Hill, R. C., & Adkins, L. C. (2001). Collinearity. In B. H. Baltagi 
>> (Ed.), A Companion to Theoretical Econometrics (p. 256-278). Oxford: 
>> Blackwell.
>>
>> * Marquardt, D. W. (1987). Collinearity and Least Squares Regression: 
>> Comment. Statistical Science, 2(1), 84-85. Available from 
>> http://projecteuclid.org/euclid.ss/1177013440
>>
>> * Montgomery, D. C., & Peck, E. A. (1982). Introduction to Linear 
>> Regression Analysis. New York, NY: John Wiley.
>>
>> * Simon, S. D., & Lesage, J. P. (1988, January). The impact of 
>> collinearity involving the intercept term on the numerical accuracy of 
>> regression. Computational Economics (formerly Computer Science in 
>> Economics and Management), 1(2), 137-152.
>>
>> * Snee, R. D., & Marquardt, D. W. (1984, May). Comment: Collinearity 
>> Diagnostics Depend on the Domain of Prediction, the Model, and the 
>> Data. The American Statistician, 38(2), 83-87.
>>
>> * Weisberg, S. (1980). Applied Linear Regression. New York, NY: John 
>> Wiley.
>>
>> * Wood, F. S. (1984, May). Comment: Effect of Centering on 
>> Collinearity and Interpretation of the Constant. The American 
>> Statistician, 38(2), 88-90.
>>
>>
>>
>> Christina Bogner schrieb:
>>> Dear list members,
>>>
>>> I'm working with both nlme and lme4 packages trying to fit linear 
>>> mixed-models to soil chemical and physical data. I know that for
>>> linear models one can calculate the condition number kappa of the
>>> model matrix to know whether the problem is well- or ill-conditioned.
>>> Does it make any sense to compute kappa on the design matrix of the
>>> fixed-effects in nlme or lme4? For comparison I fitted a simple
>>> linear model to my data and scaling some numerical predictors
>>> decreased kappa considerably. So I wonder if scaling them in the
>>> mixed-model has any advantages?
>>>
>>> Thanks a lot for your help.
>>>
>>> Christina Bogner
>>>
>>
>