hi Greg, David, and Tal,

Thank you very much for the information.

I found this in SPSS 17.0 missing value manual:

EM Method

This method assumes a distribution for the partially missing data and bases inferences
on the likelihood under that distribution. Each iteration consists of an E step and an
M step. The E step finds the conditional expectation of the “missing” data, given the
observed values and current estimates of the parameters. These expectations are then
substituted for the “missing” data. In the M step, maximum likelihood estimates of the
parameters are computed as though the missing data had been filled in. “Missing” is
enclosed in quotation marks because the missing values are not being directly filled in.
Instead, functions of them are used in the log-likelihood.

From the literatures, I got the idea that multiple imputation is a better approach. What I am thinking is, is it possible that R could output a single dataset after the whole EM or multiple imputation process, with the estimated values "filled in" for the missing values. What I am trying to get is a single data set, then I could use it for analysis (like ANOVA) and get the main effect of the whole IV instead of the main effects of each category of the IV(if multiple imputation was used). Some people suggested that sometimes the main effects of each category of IV were averaged to get the effect of the whole IV. I have not found the literature talking about this, but my guts told me this may not be a good idea since the estimates and standard error, significance... , etc can not be just simply averaged. Also, I understand that single imputation in this situation may not be appropriate.

Maybe I just asked for too much:)

Best regards,




ya

From: Greg Snow
Date: 2012-07-21 23:35
To: xinxi813
CC: r-help
Subject: Re: [R] EM for missing data
The EM algorithm does not impute missing data, rather it estimates
parameters when you have missing data (those parameters can then be
used to impute the missing values, but that is separate from the EM
algorithm).

If you create a dataset that has missing values imputed (a single
time) and then analyze that dataset as if there were no missing data
then your results will be wrong.  The better approach is multiple
imputation (and there are packages including MICE to do this) where
more than one new dataset is imputed (including error on the imputed
missing values), then each of the imputed datasets is analyzed (don't
look at the results yet, they are still each wrong), then the analyses
are combined to give a correct answer (well as correct as any
statistical procedure is, approximate is probably the better term).
Though this of course is assuming that your assumptions are
reasonable.

If SPSS really gives you a single imputed dataset after running EM for
you to analyze using other tools then my opinion of SPSS will go down.
 The reason that you probably have not found a way to do this in SAS
or R is because they are useful tools that try to not make it easy to
do the wrong thing.

On Sat, Jul 21, 2012 at 5:55 AM, ya <xinxi813@163.com> wrote:
> Hi list,
>
> I am wondering if there is a way to use EM algorithm to handle missing data and get a completed data set in R?
>
> I usually do it in SPSS because EM in SPSS kind of "fill in" the estimated value for the missing data, and then the completed dataset can be saved and used for further analysis. But I have not found a way to get the a completed data set like this in R or SAS. With Amelia or MICE, the missing data set were imputed a couple of times, and the new imputed datasets were not combined. I understand that the parameter estimation can still be done in the way of combination of estimates from each imputed data set, but it would be more convenient to have a combined dataset to do some analysis, for example, ANOVA with IVs having more than two categories. In this case, the only way to get the main effect of the whole IV is to estimate parameters in a single data set(as far as I know). If the separated imputed data sets were used, then the main effect showed in the result were for each category of the IV, respectively. I figured sometimes the readers and reviewers would like to see how bi!
>  g the effect for the whole IV instead of the effect of each category of that IV.
>
> This is one of the reasons I can not fully move to R from SPSS. So any suggestions?
>
> Thank you very much.
>
>
>
>
> ya
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.



-- 
Gregory (Greg) L. Snow Ph.D.
538280@gmail.com
	[[alternative HTML version deleted]]

