
Dear all,
Not a problem that is very specific to R, but I think that it is not only of interest to me... So I hope that someone finds the time to provide me with some clues on this probably rather basic issue.
I'm performing an analysis of experimental data with a categorical response variable. Until now I have been using a classical maximum likelihood approach. The most basic output measure to evaluate the models is thus minus twice the log likelihood, that is, the deviance. I am evaluating the statistical significance of effects by calculating the absolute difference between deviances (for nested models), which should be approximately chi-square distributed under the null hypothesis - that is the classical Wilks' test. I use bootstrapping sometimes where I suspect that the approximation is not good. But for the rest, the philosophy of the test is the same.I'm quite sure that the magnitude of effect of the experimental manipulations is highly subject-dependent. I have, in this specific case, 7 subjects; therefore, I perform and report 7 separate analyses. The problem afterwards lies in the model selection. Sometimes, the best-fitting model (either based on all-possible subset fits or a sequential procedure, with p-values, AIC or BIC based) for subject A is X1+X2, for subject B X1+X3, for C X1+X1*X2 etc. You can imaginethat this turns into a mess. I thought that evaluation of the fits of competing models could be based on a likelihood statistic that is jointly determined for all subjects, namely by aggregating (adding) deviances across subjects. In theory, if I can say that within-subject -2 log likelihood ratios to test for effects are asymptotically chi-square distributed, the sum should also be, with degrees of freedom added together. So this way I'm actually doing a kind of meta-analysisby adding deviances.(1) How defendable is this procedure? Strangely enough (or maybe not...), I didn't find anything written about a procedure like this. This leads me to believe I must be doing something very wrong... The problem, I suppose, is that I'm treating effects that are actually random, as fixed. But what are the consequences of this? Does it mean that I am making inferences about the specific sample that I have used in the experiment, that are possibly not generalizeable across the population, or is the problem more serious?(2) In the kind of situation I am facing, I often see applications of generalized linear mixed model, empirical Bayes, and the like. Somehow I think that a hierarchical Bayes model would be overkill for this. Hypothesis tests would turn into confidence regions. It might be computationally unfeasible anyway because in some of my alternative models, levels of the experimental variable are treated in a categorical way, generating a lot of free parameters that are subject to different random distributions. I'm still getting acquainted with this specific technique, so I count on your patience in case this is a stupid question: If maximum-likelihood estimation is as Bayesian inference with a flat prior on the parameters, is the meta-analysis I am trying out here then equivalent to empirical Bayes with flat priors for the random effects?
_________________________________________________________________
Hoe herken je de vrienden van je vrienden?

	[[alternative HTML version deleted]]

