[R-sig-ME] Different results for between/within groups and within group regression analyses

[Alday, Phillip]

I don't think this set of questions was ever addressed in any of the subsequent threads on this or other forums .... and so I'll give you a quick non answer of bullet points:


* "significance" should not be the driving force in whether or not an estimate is correct. While it does tell you something about the relative error in that estimate, but it tells you very little about how well that estimate describes the data or predicts future data.


* Much as it is difficult and potentially problematic to interpret a main effect in the presence of interactions (after all, by definition, an interaction implies that the main effect changes conditional on another variable), it is also problematic to intentionally exclude a known interaction to better estimate the main effect.


* However, this above advice has to be tempered by the issues related to statistical power, the bias-variance tradeoff and over- vs. underfitting -- a lot of preprints, journal articles and recent textbooks by members of this list and their collaborators have tried to address different aspects of this in the context of mixed models, but the issues are fundamentally the same for *all* statistical models. My advice is to find the parsimonious model that best describes the data (so the best model according to AIC or BIC) and look at the predictions that model makes and derive your inferences from that, instead of getting hung up on the significance of any one coefficient. The effects package can be particularly useful here, especially for plotting these things.


Phillip

On 13/01/18 23:07, Luca Danieli wrote:
Hi Phillip,

sorry if I ask you a question.
In this moment I have a 3x4x8 matrix, where '3' is the number of groups, '8' the number of tests, and '4' the levels of the potential main effect.

Following your explanation, I was before thinking that leaving interactions out of the models would give you a better approximation of the main effect. But now that I read it again, I am unsure about it.

In my case, the '4' levels are nested in each set.
If I write lmer(Score ~ pot_ME + random effects) I have no statistical significance.
If I write lmer(Score ~ pot_ME*groups + random effects) I have statistical significance for the main effect (p<0.05) and a strong interaction (p<.001).
If I write lmer(Score ~ pot_ME*groups*tests + random effects) I have no statistical significance nor interactions.

What approach is the more correct to get information about the presence of a main effect?

(My parameters are not-continuous)

Best
Luca
________________________________
From: Alday, Phillip <Phillip.Alday at mpi.nl><mailto:Phillip.Alday at mpi.nl>
Sent: 11 January 2018 15:38
To: Luca Danieli; r-sig-mixed-models at r-project.org<mailto:r-sig-mixed-models at r-project.org>
Subject: Re: [R-sig-ME] Different results for between/within groups and within group regression analyses

I'll do it myself when I get the chance in the next day or so.  :-)

Phillip

________________________________
From: Luca Danieli <mr.lucedan at hotmail.it><mailto:mr.lucedan at hotmail.it>
Sent: Thursday, January 11, 2018 10:26 AM
To: Alday, Phillip; r-sig-mixed-models at r-project.org<mailto:r-sig-mixed-models at r-project.org>
Subject: Re: [R-sig-ME] Different results for between/within groups and within group regression analyses

Thank you Phillip!

Can I add your answer to CrossValidated for future concerns by other users?

Best
Luca
________________________________
From: Alday, Phillip <Phillip.Alday at mpi.nl><mailto:Phillip.Alday at mpi.nl>
Sent: 11 January 2018 15:18
To: Luca Danieli; r-sig-mixed-models at r-project.org<mailto:r-sig-mixed-models at r-project.org>
Subject: Re: [R-sig-ME] Different results for between/within groups and within group regression analyses

By only using one group, you're changing the amount of pooling going on, which affects shrinkage and the bias-variance / over- vs. underfitting tradeoff. When you fit a model to a subset, it will generally be better at describing that subset but often worse at describing the full set / other sets. In other words, your subset model better describes the subset because it doesn't have to spend "resources" describing the other data, but of course this also means that it will tend to not describe the other data as well - it's better at the small details but worse at the big picture.

Best,
Phillip

Sent from my mobile, please excuse my brevity.

________________________________
From: Luca Danieli <mr.lucedan at hotmail.it><mailto:mr.lucedan at hotmail.it>
Sent: Thursday, January 11, 2018 10:10 AM
To: r-sig-mixed-models at r-project.org<mailto:r-sig-mixed-models at r-project.org>
Subject: [R-sig-ME] Different results for between/within groups and within group regression analyses


Dear all,

from CrossValidates I was suggested to repost my question to you, as it is a technical question about R and mixed models.
Particularly, as I have a thesis to hand in in a few weeks, I hope you are able to help me understanding some problems that I cannot figure out by myself.

In this case, I have used the function lmer() to look for an interaction between groups and then used the function predict() to plot the fits for each group on a graphic.
Then I applied the lmer() to just one of those groups (same formula, technically) and used the predict() function to plot the fits for that specific group. I was thinking to obtain the same graphic for that group type and instead I obtained two different results.

I explained the process, models and presented the plots in this post:
https://stats.stackexchange.com/questions/322608/different-results-for-between-within-groups-and-within-group-regression-analyses

Can somebody help me understand this?

Best
Luca



[[alternative HTML version deleted]]

_______________________________________________
R-sig-mixed-models at r-project.org<mailto:R-sig-mixed-models at r-project.org> mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models


	[[alternative HTML version deleted]]