[R] Off-topic: Comparing standard errors from simulation andanalytical model

[Dimitris Rizopoulos]

since you are interested especially in the standard errors, I think 
that you probably need something like a double simulation procedure, 
e.g.,

1. simulate data D[b] and "contaminate" them.

2. fit the model (with parameters \theta) using D[b], get \theta[b] 
and also compute the standard errors se.a[b] using the asymptotic 
method.

3. using \theta[b] simulate M new data sets, "contaminate" them, fit 
the model in each one, obtain \theta[m] and calculate the standard 
deviation of these estimates se.mc[b]

4. keep res[b] = (se.mc[b] - se.a[b]) / se.mc[b]

5. repeat steps 1-4 B times and calculate, e.g., a 95% CI for res 
using the sample quantiles.


of course this is going to be much more time consuming (depending on 
the choices of B and M), but I think it will give you better a picture 
of how your method performs.


I hope this helps.

Best,
Dimitris

----
Dimitris Rizopoulos
Ph.D. Student
Biostatistical Centre
School of Public Health
Catholic University of Leuven

Address: Kapucijnenvoer 35, Leuven, Belgium
Tel: +32/16/336899
Fax: +32/16/337015
Web: http://www.med.kuleuven.be/biostat/
     http://www.student.kuleuven.be/~m0390867/dimitris.htm


----- Original Message ----- 
From: "Doran, Harold" <HDoran at air.org>
To: <r-help at stat.math.ethz.ch>
Sent: Friday, September 09, 2005 4:03 PM
Subject: [R] Off-topic: Comparing standard errors from simulation 
andanalytical model


> Dear list:
>
> I'm hoping to tap in to the statistical expertise in the group,
> especially those familiar with simulation techniques. I'm finalizing 
> a
> study where I obtain standard errors from two sources. The first 
> source
> is a monte carlo simulation and the other source is an analytical 
> model
> I have developed that appears to recover the standard errors from 
> the
> simulation. All analysis are performed in R using MASS, nlme, and
> Matrix.
>
> Here is a very brief description. In the monte carlo, I first sample
> from a multivariate distribution to create data. The data are
> hypothetical student scores on an achievement test over time and the 
> aim
> is to examine what happens to standard errors under certain 
> psychometric
> conditions. The data are then "contaminated" to reflect a certain
> psychometric problem that occurs in longitudinal analyses of student
> achievement scores.
>
> These data are then analyzed using a linear model to obtain 
> parameter
> estimates. This is replicated 250 times.
>
> For example, the model equation used is
>
> Y_{ti} =  \mu + \beta \cdot t + \epsilon_{ti}
>
> So, I obtain 250 estimates of \mu and \beta. I take the standard
> deviation of these estimates to get the sampling distribution of the
> parameter (standard errors). Next, I take a single data set, 
> contaminate
> the scores, and then use the analytical approach to obtain standard
> errors. So, I end up with two sets of standards errors, those 
> obtained
> under simulated conditions and those obtained from the analytical 
> model.
>
> My question is what are the most acceptable techniques for comparing 
> the
> standard errors in order to say that the analytical approach 
> actually
> "recovers" the monte carlo standard errors? For the most part, the
> standard errors appear to be exactly the same, save rounding error.
>
> One idea I am toying with is to average the standard errors of \mu 
> and
> \beta from the simulation and then do a t-test between the two 
> standard
> errors which might be something along these lines
>
> t = (SE_{analytical} - SE_{mc} )/  \bar se
>
> Where \bar se is the average of the standard errors.
>
> But I'm not certain this is correct. Can anyone suggest a more
> appropriate method for comparing the results?
>
> Many thanks. I can also send a copy of the paper to anyone who would
> like more information or details.
>
> -Harold
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! 
> http://www.R-project.org/posting-guide.html
> 


Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm