On Fri, Mar 4, 2011 at 3:00 PM, David Duffy <davidD@qimr.edu.au> wrote:

> On Fri, 4 Mar 2011, Colin Wahl wrote:
>
>
>> The observation level effect accounts for a large portion of the variance.
>> Should I interpret this as meaning the observation level effect is
>> improving
>> the model fit substantially? If so, would I expect the standard errors to
>> decrease?
>>
>> The reason I ask is that I noticed the standard errors for the fixed
>> effects
>> in the model with the observation level effect are consistently decreased
>> by
>> ~1 or 2 percent for all fixed effects, resulting in slightly more
>> significant p values. This seems counter to your last point that this
>> "correction" should inflate the SEs by about the right amount.
>>
>> Is it still possible that I have an over fitted model? If so, how can I
>> determine if that is the case?
>>
>>
> I presume John M will say something cogent, but I am wondering more:
>
> 1) what are your key scientific questions?  Are you are more interested in
> the hypothesis testing on fixed effects, where your conclusions would seem
> to be same under either model, or do you want predictions for modelling
> populations?
>
> My interest is in testing fixed effects. My knowledge of population
modeling/simulation is limited and not the aim of my study. I suspect my
approach is likely too simple/coarse to do so adequately anyway.

My primary objective is two fold: 1. To determine this relatively simple
approach using very coarse taxonomic resolution can adequately test for
biological impairment in streams and 2. To see how tolerant/intolerant
Orders of benthic invertebates vary between watershed and riparia type.

1. Is the relative abundance of EPT (intolerant orders of
macroinvertebrates) less in disturbed watersheds compared to undisturbed
watersheds?
2. Assuming streams are disturbed by land use, does the existence of an
extant riparian corridor significantly improve macroinvertebrate
communities?

Basically: Does land use affect invertebrate communities? Do riparian
corridors help?


2) Why are you including stream:rip as a random effect, with three
> observations per level? Or are there more than 72 observations? I certainly
> can't justify why I think it is OK to have 12 random effects for stream,
> plus 72 random effects for subject, but get nervous when you also want
> another 24, but I do ;)  I think it is because I am happy to do this when
> there are strong mechanistic reasons as in genetic models, but less so when
> one is adding it here, where the fixed effect of rip was less than stellar.
>
> There are only 72 observations total. Three per site, 2 sites for each of
12 streams.

My interpretation of mixed models is still quite rudimentary, so please
forgive and correct where I need it. I have included a stream:riparia
interaction in the model for habitual rather than mechanistic reasons. I was
taught to build linear models (even with random effects) before I heard of a
mixed effect model (which has actually come to my attention surprisingly
recently, after being consistently reassured that I'll be doing a simple
anova with different error terms/F ratios). The method I used to build
linear models included all interactions that could exist. I never thought to
justify their inclusion, but understood that a small sample size will
decrease the power of any effect.

I expect %EPT to vary between streams randomly, but I also expect that %EPT
will differ randomly between riparian types within streams. It is my
understanding that using structures of random variation that exist in the
design, the model will tease the random variation and make the fixed effects
more clear. That is how I would currently justify its inclusion. I'm not
certain the argument is a good one, though. If someone were to ask me to
justify including an observation level random effect, which I interpret as
accounting for sample to sample variation, I wouldnt know where to start
(presuming "because it models for overdispersion" doesnt suffice).

3) For further model criticism, re "overfitting", I could not rely on simple
> summaries, such as you are giving us, but eyeball the predicted values for
> every level of wsh and stream, and see if they look sensible.
> On the face of it, the loglikelihood for the observation level effect model
> below is much better than for the second model.  The only way I have tripped
> up in the past is when the model is completely inappropriate for the data,
> so that likelihood ratios of that type are completely dependent on a few
> extreme observations.
>
>
The predictions are quite sensible. There are not many, if any, serious
outliers. Those samples that could be interpreted as outliers are mostly
from disturbed watersheds where the population mean was near zero (~0% or
~1-3%). In plotting actual vs. fitted values, and fitted vs. standardized
residuals, the fitted values are not particularly accurate for values near
zero. The only substantial residuals lie near zero and the term
"substantial" probably misleading.

4) If the dataset is not any larger than 72 observations, then maybe you
> should be looking at Bayesian approaches, where you could inject prior
> knowledge about the area.  I presume other people have analysed similar
> data...
>
> I was a little afraid someone might recommend this. I have virtually no
prior knowledge (pun intended?) of Bayesian approaches aside from the
discussion on this list (which I tend to skim when I dont follow...which is
often). This might be a question that could better be answered by typing
"bayesian statistics" in wikipedia, but what *kind *of prior knowledge about
the area would be used. Invertebrate communities?

>
>
>
>  Here are the glmer outputs for the model with and without the obs level
>> effect.
>>
>> The model with the observation level effect:
>> Generalized linear mixed model fit by the Laplace approximation
>> Formula: E ~ wsh * rip + (1 | stream) + (1 | stream:rip) + (1 | obs)
>>  Data: ept
>>  AIC   BIC logLik deviance
>> 284.4 309.5 -131.2    262.4
>> Random effects:
>> Groups     Name        Variance Std.Dev.
>> *obs         (Intercept) 0.30186  0.54942*
>> stream:rip (Intercept) 0.40229  0.63427
>> stream      (Intercept) 0.12788  0.35760
>> Number of obs: 72, groups: obs, 72; stream:rip, 24; stream, 12
>>
>> Fixed effects:
>>           Estimate Std. Error z value Pr(>|z|)
>> (Intercept)   -4.2906     *0.4935*  -8.694  < 2e-16 ***
>> wshd           -2.0557     *0.7601*  -2.705  0.00684 **
>> wshf             3.3575     *0.6339*   5.297 1.18e-07 ***
>> wshg            3.3923     *0.7486*   4.531 5.86e-06 ***
>> ripN              0.1425     *0.6323*   0.225  0.82165
>> wshd:ripN     0.3708     *0.9682*   0.383  0.70170
>> wshf:ripN     -0.8665     *0.8087*  -1.071  0.28400
>> wshg:ripN    -3.1530    * 0.9601*  -3.284  0.00102 **
>>
>> Model without the Observation level effect:
>> Generalized linear mixed model fit by the Laplace approximation
>> Formula: E ~ wsh * rip + (1 | stream) + (1 | stream:rip)
>>  Data: ept
>>  AIC BIC logLik deviance
>> 754.3 777 -367.2    734.3
>> Random effects:
>> Groups     Name        Variance Std.Dev.
>> stream:rip (Intercept) 0.48908  0.69934
>>  stream     (Intercept) 0.18187  0.42647
>> Number of obs: 72, groups: stream:rip, 24; stream, 12
>>
>> Fixed effects:
>>           Estimate Std. Error z value Pr(>|z|)
>> (Intercept)  -4.28529    *0.50575*  -8.473  < 2e-16 ***
>> wshd          -2.06605    *0.77357*  -2.671  0.00757 **
>> wshf            3.36248    *0.65118 *  5.164 2.42e-07 ***
>> wshg           3.30175    *0.76962*   4.290 1.79e-05 ***
>> ripN             0.07063    *0.61930*   0.114  0.90920
>> wshd:ripN    0.60510    *0.94778*   0.638  0.52319
>> wshf:ripN    -0.80043    *0.79416*  -1.008  0.31350
>> wshg:ripN   -2.78964    *0.94336*  -2.957  0.00311 **
>>
>>
> --
> | David Duffy (MBBS PhD)                                         ,-_|\
> | email: davidD@qimr.edu.au  ph: INT+61+7+3362-0217 fax: -0101  /     *
> | Epidemiology Unit, Queensland Institute of Medical Research   \_,-._/
> | 300 Herston Rd, Brisbane, Queensland 4029, Australia  GPG 4D0B994A v
>



-- 
Colin Wahl
Department of Biology
Western Washington University
Bellingham WA, 98225
ph: 360-391-9881

	[[alternative HTML version deleted]]


