[R-sig-ME] suggestions for estimating confidence intervals with only 1.5 GB of ram?
Darren Norris
doon75 at hotmail.com
Thu Mar 31 23:58:01 CEST 2011
Dear all,
Doug has previously explained how to approximate confidence intervals
for parameters
( <AANLkTin+CRAH6EoeK9+-BS4+cwRr6sO-r4dgkRinqOjy at mail.gmail.com
<mailto:AANLkTin+CRAH6EoeK9+-BS4+cwRr6sO-r4dgkRinqOjy at mail.gmail.com>> )
and Ben Bolker (GLMM wiki - http://glmm.wikidot.com/faq) and Mike
Laurance (ezPredict in package "ez") have made functons / shared code
for calculating and plotting fitted values from lmer models and intervals.
Just some examples of what I have found - Fantastic! A big thank you
for all the time taken. What more could I need?
The problem and reason for this email ( apologies for the length but I
found it hard to explain) :
I would like to obtain (then plot) the fitted (?predicted - sorry never
know correct term) response values and in an ideal world 95% confidence
(or prediction) intervals for a lmer model.
Using Bens code example as adapted in Mikes "ezPredict" I run out of
memory when trying to estimate confidence intervals (I think Ben may
have given a health warning about this issue but I can't find the
email). This is in no way a complaint / critisim but simply a request
for help to make the best use of the resources at my disposal.
Why do I need a confidence interval? Because the "upper" values are
biologically important - they will represent the minimum abundance of a
turtle species.
The dataframe (link to data and code below) has 333 rows. I am trying to
predict the response values with 7 factors (2 continuous and 5
categorical - total of 24 parameters). In an ideal world I would get an
extra 3 columns of prediction, lower interval, upper interval to add to
my dataframe.
Is my model appropriate for deriving confidence intervals - i.e. even if
I had 64 bit linux with 1TB of ram would it work?
Can anyone suggest alternatives for how I can obtain predictions and 95%
confidence interval for this data set with a limit of 1.5GB of memory?
Thank you for any suggestions, more details below.
Darren
My windows 32 bit laptop has a memory limit of approx1.5 GB for R
("sessionInfo()" below), I do not have finances to buy a new laptop.
The 64 bit windows 7 desktop in my laboratory has 8GB of ram for R. No
more money to buy a new desktop and as is for use in a group, even
installing a virtual linux distro is not an option.
My surprise is that even on the desktop with 8GB ram I ran out of memory.
I feel that I must be doing something wrong or "missing a trick" to
obtain predictions and intervals for my model which is based on a
relatively small amount of data.
I am aware that even with my "small" dataset the |"expand.grid|"
function returns a "big amount" of data.
Is there any way to make the workflow / use of these functions (that Ben
kindly shared here http://glmm.wikidot.com/faq section "Predictions
and/or confidence (or prediction) intervals on predictions") work more
efficiently?
The data is available at this link as a R workspace
"DarrensSpace.RData" (size is approx 70 kb):
http://cid-f0a9fa3480208398.office.live.com/self.aspx/lmeData/DarrensSpace.RData
#### the random effect (1|yearMonth) is used to model serial temporal
autocorrelation,
#### data is observational & ecological so unbalanced
#### "yall " is abundance of the turtle species of interest
library("lme4")
fmer2f<-lmer(yall~yearSeason+sun+total_precip_trip+mean_temp_trip+surveyArea+obscat+hour_period+(1|yearMonth),REML=FALSE,data=df.p)
library("ez")
er_pred<- ezPredict(fit = fmer2f) ## runs out of memory here
##### session info for the laptop I use
R version 2.12.1 (2010-12-16)
Platform: i386-pc-mingw32/i386 (32-bit)
locale:
[1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United
Kingdom.1252 LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C LC_TIME=English_United
Kingdom.1252
attached base packages:
[1] grid splines stats graphics grDevices utils
datasets methods base
other attached packages:
[1] ez_3.0-0 lme4_0.999375-37 Matrix_0.999375-46
lattice_0.19-13 stringr_0.4 ggplot2_0.8.9 proto_0.3-8
[8] reshape_0.8.3 plyr_1.4 reshape2_1.1
car_2.0-9 survival_2.36-2 nnet_7.3-1 MASS_7.3-9
loaded via a namespace (and not attached):
[1] nlme_3.1-97 stats4_2.12.1
More information about the R-sig-mixed-models
mailing list