[R-sig-ME] lme capable of running with missing data?
Andrew Slack-Smith
aslacksmith2 at gmail.com
Fri Feb 3 18:18:35 CET 2012
Charles
As you suggest SAS will run the data. I just put it into SAS and all OK.
SAS uses the Satterthwaite "degrees of freedom method". If that helps at
all.
>From the web page http://glmm.wikidot.com/faq I grabbed the following which
may relate to your problem.
- Other df approximation schemes that have been suggested (Satterthwaite,
Kenward-Roger, etc.) would apparently be fairly hard to implement in
lme4/nlme, both because of a difference in notational framework and because
naive approaches would be computationally difficult in the case of large
data sets. (The Kenward-Roger approach has now been implemented in the
pbkrtest package (as KRmodcomp): note that it was derived for LMMs and is
not necessarily applicable to GLMMs!)
- When the data are not classical (crossed, unbalanced, R-side effects), we
might still guess that the deviances etc. are approximately F-distributed
but that we don't know the real degrees of freedom - this is what the
Satterthwaite, Kenward-Roger, Fai-Cornelius, etc. approximations are
supposed to do.
All the best
Andrew
-----Original Message-----
From: r-sig-mixed-models-bounces at r-project.org
[mailto:r-sig-mixed-models-bounces at r-project.org] On Behalf Of Kevin Wright
Sent: Saturday, 4 February 2012 3:58 AM
To: Charles Determan Jr
Cc: r-sig-mixed-models at r-project.org
Subject: Re: [R-sig-ME] lme capable of running with missing data?
Charles,
Here's a simple thought example. Use a piece of graph paper (or just a
simple sketch). Write the following letters at the coordinates specified:
A1 (1,1)
A2 (3,2)
B1 (1,2)
B2 (missing)
Draw a line from A1 to A2. Imagine a line from B1 to the missing value of
B2.
By looking at this, you could calculate an overall mean for the A factor.
You could also estimate an overall mean for the B factor, if you assume the
lines are parallel. This is what happens with fixed effects, as in lme (
... A + B, ...).
But, when you specify lme(... A*B, ...) which is the same as lme(... A + B
+ A:B, ...), you are essentially saying to the computer, "The A1-A2 and
B1-B2 lines are not parallel, but please give me an estimate of the slope of
the B1-B2 line."
Could _you_ draw the B1-B2 line? No. Neither can lme.
It's okay that B2 is missing if you don't want to fit an interaction, but
when B2 is missing, there is no way to estimate an interaction (non-parallel
slope).
Kevin
On Fri, Feb 3, 2012 at 10:18 AM, Charles Determan Jr
<deter088 at umn.edu>wrote:
> After the data is input, and factors are assigned,
>
> model=lme(arginine~group*time*survival, random=~1|subj, method="REML",
> data=x)
>
> Error in MEEM(object, conLin, control$niterEM) : Singularity in
> backsolve at level 0, block 1
>
>
>
> On Fri, Feb 3, 2012 at 10:01 AM, Kevin Wright <kw.stat at gmail.com> wrote:
>
>> Providing the data is not a "reproducible example". Complete data
>> and R-code are helpful.
>>
>> Kevin
>>
>>
>>
>> On Fri, Feb 3, 2012 at 9:31 AM, Charles Determan Jr
<deter088 at umn.edu>wrote:
>>
>>> Here is the dataset, everything should be run as a factor except 'met'
>>> which is numeric. Thanks for the assistance,
>>>
>>> time group survival subj met
>>> 1 1 2 1 2 1.3954
>>> 2 2 2 1 2 1.8063
>>> 3 3 2 1 2 1.3684
>>> 4 4 2 1 2 2.0046
>>> 5 5 2 1 2 1.0334
>>> 6 6 2 1 2 0.3644
>>> 7 7 2 1 2 0.4819
>>> 8 8 2 1 2 1.4558
>>> 9 9 2 1 2 0.9718
>>> 10 1 1 2 5 0.7771
>>> 11 2 1 2 5 1.2439
>>> 12 1 2 2 8 1.0980
>>> 13 2 2 2 8 0.9511
>>> 14 1 2 1 9 1.0534
>>> 15 2 2 1 9 1.7279
>>> 16 3 2 1 9 1.4904
>>> 17 4 2 1 9 1.2737
>>> 18 5 2 1 9 0.8929
>>> 19 6 2 1 9 0.5828
>>> 20 7 2 1 9 0.3260
>>> 21 8 2 1 9 1.0373
>>> 22 9 2 1 9 0.9624
>>> 23 1 2 2 10 1.1391
>>> 24 2 2 2 10 1.3945
>>> 25 3 2 2 10 0.9414
>>> 26 4 2 2 10 1.1152
>>> 27 5 2 2 10 0.8222
>>> 28 6 2 2 10 0.4417
>>> 29 7 2 2 10 0.4126
>>> 30 1 1 1 12 1.3024
>>> 31 2 1 1 12 1.1811
>>> 32 3 1 1 12 0.9379
>>> 33 4 1 1 12 1.3000
>>> 34 5 1 1 12 1.2977
>>> 35 6 1 1 12 0.4949
>>> 36 7 1 1 12 0.5238
>>> 37 8 1 1 12 1.3862
>>> 38 1 1 1 16 1.2259
>>> 39 2 1 1 16 0.8681
>>> 40 3 1 1 16 1.2645
>>> 41 4 1 1 16 0.7316
>>> 42 5 1 1 16 0.6648
>>> 43 6 1 1 16 0.9671
>>> 44 7 1 1 16 1.0131
>>> 45 8 1 1 16 1.1762
>>> 46 9 1 1 16 0.8776
>>> 47 1 2 2 18 1.1231
>>> 48 2 2 2 18 1.2133
>>> 49 3 2 2 18 1.2005
>>> 50 4 2 2 18 0.7198
>>> 51 5 2 2 18 0.6620
>>> 52 6 2 2 18 0.5908
>>> 53 7 2 2 18 0.3945
>>> 54 1 2 2 19 0.7852
>>> 55 2 2 2 19 0.6758
>>> 56 3 2 2 19 0.5246
>>> 57 4 2 2 19 0.5263
>>> 58 1 2 2 20 1.2284
>>> 59 2 2 2 20 0.7017
>>> 60 1 2 1 23 0.9604
>>> 61 2 2 1 23 0.7977
>>> 62 3 2 1 23 1.2267
>>> 63 4 2 1 23 1.3857
>>> 64 5 2 1 23 0.9486
>>> 65 6 2 1 23 0.3571
>>> 66 7 2 1 23 0.3134
>>> 67 8 2 1 23 1.9984
>>> 68 9 2 1 23 0.4837
>>> 69 1 1 1 24 1.1793
>>> 70 2 1 1 24 1.3883
>>> 71 3 1 1 24 2.1080
>>> 72 4 1 1 24 0.8810
>>> 73 5 1 1 24 0.8825
>>> 74 6 1 1 24 0.4124
>>> 75 7 1 1 24 0.5270
>>> 76 8 1 1 24 1.9003
>>> 77 9 1 1 24 1.4344
>>> 78 1 1 1 27 1.1905
>>> 79 2 1 1 27 1.1033
>>> 80 3 1 1 27 1.4976
>>> 81 4 1 1 27 1.9018
>>> 82 5 1 1 27 0.5815
>>> 83 6 1 1 27 0.4428
>>> 84 7 1 1 27 0.4728
>>> 85 8 1 1 27 1.6309
>>> 86 9 1 1 27 0.4054
>>> 87 1 1 1 28 0.9538
>>> 88 2 1 1 28 0.7796
>>> 89 3 1 1 28 1.7906
>>> 90 5 1 1 28 0.4715
>>> 91 6 1 1 28 0.4214
>>> 92 7 1 1 28 0.4120
>>> 93 8 1 1 28 1.3111
>>> 94 9 1 1 28 0.3677
>>> 95 1 1 2 1 1.3853
>>> 96 2 1 2 1 1.5966
>>> 97 3 1 2 1 1.4542
>>> 98 4 1 2 1 1.3084
>>> 99 5 1 2 1 1.2826
>>> 100 6 1 2 1 0.6835
>>> 101 7 1 2 1 0.9709
>>> 102 1 1 1 3 1.3175
>>> 103 2 1 1 3 0.7792
>>> 104 3 1 1 3 1.8763
>>> 105 5 1 1 3 1.4633
>>> 106 6 1 1 3 0.0735
>>> 107 7 1 1 3 0.5612
>>> 108 8 1 1 3 1.3777
>>> 109 9 1 1 3 0.3810
>>> 110 1 1 2 4 1.3486
>>> 111 1 1 1 6 1.2635
>>> 112 2 1 1 6 0.7572
>>> 113 3 1 1 6 1.5011
>>> 114 5 1 1 6 0.6873
>>> 115 6 1 1 6 0.3778
>>> 116 7 1 1 6 0.4231
>>> 117 8 1 1 6 1.3817
>>> 118 9 1 1 6 0.5850
>>> 119 1 2 2 7 0.7362
>>> 120 2 2 2 7 0.5495
>>> 121 3 2 2 7 0.7621
>>> 122 4 2 2 7 0.8421
>>> 123 5 2 2 7 1.0438
>>> 124 6 2 2 7 0.9802
>>> 125 7 2 2 7 0.5627
>>> 126 1 1 1 11 1.5575
>>> 127 2 1 1 11 2.1356
>>> 128 3 1 1 11 1.3575
>>> 129 4 1 1 11 1.3056
>>> 130 5 1 1 11 0.8144
>>> 131 6 1 1 11 0.5876
>>> 132 7 1 1 11 0.4104
>>> 133 9 1 1 11 0.4942
>>> 134 1 2 1 13 1.0046
>>> 135 2 2 1 13 0.8805
>>> 136 3 2 1 13 0.7685
>>> 137 4 2 1 13 0.8786
>>> 138 5 2 1 13 1.4249
>>> 139 6 2 1 13 0.5339
>>> 140 7 2 1 13 0.5480
>>> 141 8 2 1 13 2.6369
>>> 142 9 2 1 13 1.7159
>>> 143 1 2 1 14 0.7161
>>> 144 2 2 1 14 0.3968
>>> 145 3 2 1 14 0.8142
>>> 146 4 2 1 14 0.6140
>>> 147 5 2 1 14 0.6585
>>> 148 6 2 1 14 0.7176
>>> 149 7 2 1 14 0.6613
>>> 150 8 2 1 14 1.6494
>>> 151 9 2 1 14 0.3903
>>> 152 1 1 1 15 1.4357
>>> 153 2 1 1 15 1.4772
>>> 154 3 1 1 15 1.3156
>>> 155 4 1 1 15 0.9654
>>> 156 5 1 1 15 1.2709
>>> 157 6 1 1 15 0.9330
>>> 158 7 1 1 15 0.3515
>>> 159 8 1 1 15 1.6801
>>> 160 9 1 1 15 0.3584
>>> 161 1 2 2 17 0.8077
>>> 162 2 2 2 17 0.7560
>>> 163 1 1 1 21 1.1890
>>> 164 2 1 1 21 0.9631
>>> 165 3 1 1 21 0.9753
>>> 166 4 1 1 21 0.9519
>>> 167 5 1 1 21 0.6348
>>> 168 6 1 1 21 0.8516
>>> 169 7 1 1 21 0.2366
>>> 170 8 1 1 21 1.0440
>>> 171 9 1 1 21 0.5360
>>> 172 1 2 1 22 1.0747
>>> 173 2 2 1 22 0.6451
>>> 174 3 2 1 22 0.8408
>>> 175 5 2 1 22 0.8730
>>> 176 6 2 1 22 0.3594
>>> 177 7 2 1 22 0.3019
>>> 178 9 2 1 22 1.2053
>>> 179 1 2 2 25 0.4654
>>> 180 2 2 2 25 0.3024
>>> 181 3 2 2 25 0.7525
>>> 182 4 2 2 25 0.7808
>>> 183 5 2 2 25 0.6294
>>> 184 6 2 2 25 0.3016
>>> 185 7 2 2 25 0.3223
>>> 186 1 2 1 26 0.5363
>>> 187 2 2 1 26 0.2279
>>> 188 3 2 1 26 0.4756
>>> 189 4 2 1 26 0.6644
>>> 190 5 2 1 26 0.6631
>>> 191 6 2 1 26 0.3419
>>> 192 7 2 1 26 0.4188
>>> 193 8 2 1 26 0.3199
>>> 194 9 2 1 26 0.2889
>>> 195 1 1 2 29 1.2765
>>> 196 2 1 2 29 1.0653
>>> 197 3 1 2 29 1.5607
>>> 198 1 1 1 30 0.8641
>>> 199 2 1 1 30 0.9250
>>> 200 3 1 1 30 1.0887
>>> 201 4 1 1 30 0.5537
>>> 202 5 1 1 30 0.7930
>>> 203 6 1 1 30 0.3960
>>> 204 7 1 1 30 0.3917
>>> 205 8 1 1 30 1.2687
>>> 206 9 1 1 30 0.5328
>>> 207 1 2 1 31 1.0765
>>> 208 2 2 1 31 0.8778
>>> 209 3 2 1 31 0.8228
>>> 210 4 2 1 31 1.2017
>>> 211 5 2 1 31 1.1787
>>> 212 6 2 1 31 0.4037
>>> 213 7 2 1 31 0.2625
>>> 214 8 2 1 31 2.2690
>>> 215 9 2 1 31 0.4423
>>> 216 1 1 2 32 1.2880
>>> 217 2 1 2 32 0.8537
>>>
>>> On Fri, Feb 3, 2012 at 9:25 AM, Baldwin, Jim -FS
>>> <jbaldwin at fs.fed.us>
>>> wrote:
>>>
>>> > I think the only way to resolve this is to provide a specific example.
>>> >
>>> > Jim Baldwin
>>> > Station Statistician
>>> > USDA Forest Service
>>> > Albany, California
>>> >
>>> > -----Original Message-----
>>> > From: r-sig-mixed-models-bounces at r-project.org [mailto:
>>> > r-sig-mixed-models-bounces at r-project.org] On Behalf Of Charles
>>> Determan Jr
>>> > Sent: Friday, February 03, 2012 7:18 AM
>>> > To: Thompson,Paul; r-sig-mixed-models at r-project.org
>>> > Subject: Re: [R-sig-ME] lme capable of running with missing data?
>>> >
>>> > So, is there a way in which I can alter the design matrix so the
>>> > mixed model will work or is this something that can only be done
>>> > in SAS currently? The output from the SAS run did provide Type
>>> > III fixed
>>> effect
>>> > test values.
>>> >
>>> > On Fri, Feb 3, 2012 at 9:14 AM, Thompson,Paul <
>>> > Paul.Thompson at sanfordhealth.org> wrote:
>>> >
>>> > > That's interesting. SAS uses the sweep approach (it was in fact
>>> > > devised by Goodnight). The method used in construction of
>>> > > various types of SS does allow you to estimate when cells are
>>> > > missing. I
>>> would
>>> > > wonder if Type II SS can be done. Type III (despite the
>>> > > incorrect statement that they are
>>> > > illegitimate) and Type IV would work fine. ****
>>> > >
>>> > > ** **
>>> > >
>>> > > It's really an issue of the manner in which the design matrix is
>>> > > contructed.****
>>> > >
>>> > > ** **
>>> > >
>>> > > *From:* Charles Determan Jr [mailto:deter088 at umn.edu]
>>> > > *Sent:* Friday, February 03, 2012 8:36 AM
>>> > > *To:* Thompson,Paul; r-sig-mixed-models at r-project.org
>>> > > *Subject:* Re: [R-sig-ME] lme capable of running with missing
>>> > > data?****
>>> > >
>>> > > ** **
>>> > >
>>> > > Thank you Paul, I do appreciate your response and especially
>>> > > your
>>> time.
>>> > > The reason I am so persistent is that I know the prior data I
>>> > > posted was run in SAS (however I don't have the exact coding
>>> > > although I do know it was done with PROC MIXED with an
>>> > > unstructured covariance structure and REML estimation method)
>>> > > and it provided all the interactions. As such, I have scoured
>>> > > the web and literature as to how this could be done with the
>>> > > missing data (timepoints as a result of survival). Perhaps this
>>> > > simply has not yet been done in R and I
>>> am
>>> > > stuck for the time being. None-the-less, I want to be certain
>>> before I
>>> > give up on running this type of analysis in R.
>>> > >
>>> > > Thanks again,****
>>> > >
>>> > > On Fri, Feb 3, 2012 at 8:26 AM, Thompson,Paul <
>>> > > Paul.Thompson at sanfordhealth.org> wrote:****
>>> > >
>>> > > Charles:
>>> > >
>>> > > I did suggest the use of specific contrasts to do the analysis
>>> > > with missing cells. I played around, and just have to admit that
>>> > > this is not possible. I tried to use standard construction
>>> > > techniques to produce main effects using contrast coding, and
>>> > > then multiply those
>>> to
>>> > > produce interactions. This does not work. It may be possible to
>>> > > use orthonormalization and the sweep operator to produce a
>>> > > consistent estimator, but I ran out of time to work on this.
>>> > >
>>> > > What you can do is convert the design to a single factor, and do
>>> > > the analysis with specific contrasts, recognizing that this will
>>> > > not enable you to get to specific things like interaction
>>> > > effects. To understand why, consider the situation with a 2 x 2,
>>> > > where one cell
>>> is
>>> > entirely missing.
>>> > > You have lost 1 df for the design, and the interaction is
>>> > > entirely
>>> > missing.
>>> > > You can estimate and test specific contrasts, but you can't even
>>> > > really test the A factor or the B factor. If Cell(2,2) is
>>> > > missing,
>>> you
>>> > > can test Cell (1,1) v Cell(1,2) and you can test Cell (2,1) v
>>> > > Cell (1,1), but neither of these is the test of the "main
>>> > > effect" of A or B. When you have larger designs with 2 or 3
>>> > > factors, the comparisons again have fewer df than should be
>>> > > encountered. This means that the interactions are not defined
properly.
>>> > >
>>> > > Do you NEED the interactions for theoretical purposes, or are
>>> > > they there simply for completedness? Are the cells missing due
>>> > > to your design or due to happenstance?
>>> > >
>>> > > It is the case that fractional factorial designs eliminate cells
>>> > > from the design to estimate main effects and losing the ability
>>> > > to
>>> estimate
>>> > > interactions. So, missing cells, when planned for appropriately,
>>> > > can result in appropriate analysis. I am not sure how to run
>>> > > mixed models with fractional factorials, however.****
>>> > >
>>> > >
>>> > > -----Original Message-----
>>> > > From: r-sig-mixed-models-bounces at r-project.org [mailto:
>>> > > r-sig-mixed-models-bounces at r-project.org] On Behalf Of Charles
>>> > > Determan Jr
>>> > > Sent: Friday, February 03, 2012 7:06 AM
>>> > > To: r-sig-mixed-models at r-project.org
>>> > > Subject: [R-sig-ME] lme capable of running with missing data?
>>> > >
>>> > > Greetings,
>>> > >
>>> > > Some of you may recognize my name from a few related posts but I
>>> > > just have general question that perhaps can be clarified. I
>>> > > have read several times that 'lme' and 'lmer' are techniques
>>> > > capable of running data sets with missing values. Is this true?
>>> > > I have put up similar posts where when I try to run a two or
>>> > > three way interaction mixed model I get an error of
>>> > > singularities or X'X not positive. Does the data set need to be
>>> > > formatted in some way where the mixed model can
>>> be
>>> > run with all interactions?
>>> > > Furthermore, if the missing values are 'not missing at random'
>>> > > is there another method to follow for generating the mixed
>>> > > model? I am just confused why I see posts that lme can be run
>>> > > when data is
>>> missing.
>>> > >
>>> > > Regards,****
>>> > >
>>> > > [[alternative HTML version deleted]]
>>> > >
>>> > > _______________________________________________
>>> > > R-sig-mixed-models at r-project.org mailing list
>>> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>> > >
>>> > >
>>> --------------------------------------------------------------------
>>> --
>>> > > - Confidentiality Notice: This e-mail message, including any
>>> > > attachments, is for the sole use of the intended recipient(s)
>>> > > and may contain privileged and confidential information. Any
>>> > > unauthorized review, use, disclosure or distribution is
>>> > > prohibited. If you are
>>> not
>>> > > the intended recipient, please contact the sender by reply
>>> > > e-mail and destroy all copies of the original message.****
>>> > >
>>> > > ** **
>>> > >
>>> > >
>>> > >
>>> --------------------------------------------------------------------
>>> --
>>> > > - Confidentiality Notice: This e-mail message, including any
>>> > > attachments, is for the sole use of the intended recipient(s)
>>> > > and may contain privileged and confidential information. Any
>>> > > unauthorized review, use, disclosure or distribution is
>>> > > prohibited. If you are not the intended recipient, please
>>> > > contact the sender by reply e-mail and destroy all copies of the
original message.
>>> > >
>>> >
>>> > [[alternative HTML version deleted]]
>>> >
>>> >
>>> >
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-mixed-models at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>
>>
>>
>>
>> --
>> Kevin Wright
>>
>>
>
--
Kevin Wright
[[alternative HTML version deleted]]
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
More information about the R-sig-mixed-models
mailing list