[R] How to perform statistical test with multiple variables

Neha gupta neh@@bo|ogn@90 @end|ng |rom gm@||@com
Mon Feb 28 23:28:29 CET 2022


Thank you Jim

I want to evaluate if there is a significant difference between several ML
models I have just run. Evaluation metric is Mean Absolute Error (MAE).
Values in brackets are MAE values of a particular model (e.g. RF) for
different folds of k-fold CV. I found that 'a' performed better (in terms
of MAE score) than the other models i.e. b, c, d, e.

But is there any significant difference ? I have used Wilcoxon test like
below:

wilcox.test(a, b, paired = T, conf.int = T, conf.level = 0.95) ,
then wilcox.test(a, c, paired = T, conf.int = T, conf.level = 0.95),
wilcox.test(a, d, paired = T, conf.int = T, conf.level = 0.95)   and so on,
I compared the best model with all the other models for statistical
differences.

Can I do a test which compares it in just one time (and one step) and not
repeating the wilcoxon test to compare the best model with all others.



a= c(0.3, 0.5, ..... 0.4)
b= c(0.4, 0.6,...... 0.2)
....
..
f= c(0.1, 0.4, .....0.5)

On Mon, Feb 28, 2022 at 11:14 PM Jim Lemon <drjimlemon using gmail.com> wrote:

> Hi neha,
> Ouch! I've just been run over by homework. What you are probably being
> asked to do is an "omnibus test", but I can't figure out what you are
> actually testing. Maybe that phrase will help you in _your_ search for
> the answer.
>
> Jim
>
> On Tue, Mar 1, 2022 at 8:56 AM Neha gupta <neha.bologna90 using gmail.com>
> wrote:
> >
> > Hello everyone,
> >
> > I have 'mean absolute error' values from different folds of CV of 6 ML
> > models..
> > a= c(0.3, 0.5, ..... 0.4)
> > b= c(0.4, 0.6,...... 0.2)
> > ....
> > ..
> > f= c(0.1, 0.4, .....0.5)
> > Now algorithm 'a' performed better than all the others (in terms of mean
> > absolute error scores) and then I used wilcoxon signed rank test to
> compare
> > 'a' with all other models for significance difference.
> >
> > Now, I have been asked to compare all the models with all others for
> > significant differences at once and not compare one with the other, how
> can
> > I do it, plz guide me? At this time, I just know about the Wilcoxon
> signed
> > rank test.
> > Regards
> >
> >         [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]



More information about the R-help mailing list