[RsR] minimum sample size for the robust counterpart of the t-test #2

Fri Jun 17 17:55:57 CEST 2011

Dear Richard,

Since we did not quite cover your specific case, I ran another small
simulation. See the attached file. It is basically the simulation
study of our paper, but for models having only an intercept. I hope I
did not overlook anything when I did this. There was a lot going on
today... I apologize for the overloaded plots. I guess Figures 4, 7
and 8 are the most interesting figures.

As Rand already stated, the asymmetric error distributions are a
problem: all the methods perform quite badly. Otherwise, the levels of
the tests are pretty much ok (even for OLS, i.e., t-test). But of
course, the power will be pretty bad. In numbers, for n = 5 you will
have approximately the correct level (+/- 2%), but a power of about
40% only for an effect size of 1 (10% for an effect size of 0.4). And
this does not really depend on which method you are using.

To conclude, I would recommend to use lmrob from robustbase with the
argument setting="KS2011".

I hope this helps,

Manuel

On Thu, Jun 16, 2011 at 8:19 PM, Richard Friedman
<friedman using cancercenter.columbia.edu> wrote:
> Rand,
>
>        Thanks, I know very little about robust methods. I am interested in
> whether rlm can be used in its default
> state or if I have to tearn much more to do use the methods correctly.
>
> Best wishes,
> Rich
>
> On Jun 16, 2011, at 2:14 PM, Rand Wilcox wrote:
>
>> When dealing with M-estimators and the goal is to compute confidence
>> intervals, one thing you have to be careful about is skewed distributions.
>> Have not encountered any non-bootstrap method that performs well in
>> simulations where the confidence interval is based on an estimate of the
>> standard error. Just how symmetric the distribution must be seems unclear.
>> What works better is a percentile bootstrap method, even with fairly small
>> sample sizes. This is why the methods in my book focus on bootstrap
>> techniques when dealing with M-estimators.
>>
>>
>> However, have not yet seen the Koller and Stahel paper. Maybe this problem
>> has been addressed.
>>
>> Rand
>>
>> Rand Wilcox
>> Professor
>> Dept of Psychology
>> USC
>> Los Angeles, CA 90089-1061
>>
>> FAX: 213-746-9082
>> For information about statistics books and software, see
>> http://www-rcf.usc.edu/~rwilcox/
>> as well as
>> http://college.usc.edu/labs/rwilcox/home
>>
>> ----- Original Message -----
>> From: Richard Friedman <friedman using cancercenter.columbia.edu>
>> Date: Thursday, June 16, 2011 9:02 am
>> Subject: Re: [RsR] minimum sample size for the robust counterpart of the
>> t-test #2
>> To: Rand Wilcox <rwilcox using usc.edu>, r-sig-robust using r-project.org
>>
>>> Dear Rand (and List),
>>>
>>>        I read the relevant sections of your book and while informative it
>>> did not answer my question
>>> directly as best I can see. I will restate the question more
>>> explicitly:
>>> A robust analog of the two sample  t-test is performed with the rlm
>>> function with the default parameters of
>>> the Huber method with K=1.345. Is there a minimum sample size for
>>> which it should be trusted?
>>> are 5 samples enough? 10 samples?
>>>
>>> If this question does not have a simple answer please let me know.
>>>
>>> Thanks and best wishes,
>>> Rich
>>>
>>>
>>> On Jun 15, 2011, at 3:19 PM, Rand Wilcox wrote:
>>>
>>>> There is general information about sample sizes and p-values, when
>>>
>>> using robust analogs of t, in my 2005 book (Introduction to Robust
>>> Estimation and Hypothesis Testing, Academic Press) .
>>>>
>>>> (A third edition will be out early in 2012. )
>>>>
>>>> Hope this helps.
>>>>
>>>> Rand
>>>>
>>>> Rand Wilcox
>>>> Professor
>>>> Dept of Psychology
>>>> USC
>>>> Los Angeles, CA 90089-1061
>>>>
>>>> FAX: 213-746-9082
>>>> For information about statistics books and software, see
>>>
>>> http://www-rcf.usc.edu/~rwilcox/
>>>>
>>>> as well as
>>>> http://college.usc.edu/labs/rwilcox/home
>>>>
>>>> ----- Original Message -----
>>>> From: Richard Friedman <friedman using cancercenter.columbia.edu>
>>>> Date: Wednesday, June 15, 2011 12:11 pm
>>>> Subject: [RsR] minimum sample size for the robust counterpart of
>>>
>>> the t-test
>>>>
>>>> To: r-sig-robust using r-project.org
>>>>
>>>>> Dear List,
>>>>>
>>>>>        I am a beginner in the use of robust methods. Is there a minimum
>>>>> sample size
>>>>> for which the robust analog of a two sample t-test using rlm with
>>>>> default parameters and categorical
>>>>> explanatory variables may be trusted to yield reliable p-values?
>>>>> Is so, can you please point me at a reference which treats this
>>>>> problem.
>>>>> Thanks and best wishes,
>>>>> Rich
>>>>> ------------------------------------------------------------
>>>>> Richard A. Friedman, PhD
>>>>> Associate Research Scientist,
>>>>> Biomedical Informatics Shared Resource
>>>>> Herbert Irving Comprehensive Cancer Center (HICCC)
>>>>> Lecturer,
>>>>> Department of Biomedical Informatics (DBMI)
>>>>> Educational Coordinator,
>>>>> Center for Computational Biology and Bioinformatics (C2B2)/
>>>>> National Center for Multiscale Analysis of Genomic Networks (MAGNet)
>>>>> Room 824
>>>>> Irving Cancer Research Center
>>>>> Columbia University
>>>>> 1130 St. Nicholas Ave
>>>>> New York, NY 10032
>>>>> (212)851-4765 (voice)
>>>>> friedman using cancercenter.columbia.edu
>>>>> http://cancercenter.columbia.edu/~friedman/
>>>>>
>>>>> I am a Bayesian. When I see a multiple-choice question on a test
>>>>> and I don't
>>>>> know the answer I say "eeney-meaney-miney-moe".
>>>>>
>>>>> Rose Friedman, Age 14
>>>>>
>>>>> _______________________________________________
>>>>> R-SIG-Robust using r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-robust
>>>>>
>>>
>>>
>
> _______________________________________________
> R-SIG-Robust using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-robust
>

-- 
Manuel Koller <koller using stat.math.ethz.ch>
Seminar für Statistik, HG G 18, Rämistrasse 101
ETH Zürich  8092 Zürich  SWITZERLAND
phone: +41 44 632-4673 fax: ...-1228
http://stat.ethz.ch/people/kollerma/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: intercept_only.pdf
Type: application/pdf
Size: 921710 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-robust/attachments/20110617/8df4fdd9/attachment.pdf>