3
$\begingroup$

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to split a dataset into train and test sets. Then, I've randomly selected small k samples (10, 20, 30, ... 100) from the train set and used them to train the classifiers and used the test set for testing. I've replicated this 5 times to make sure I got some reliable results.

Now, I want to evaluate the results. Any suggestions on a statistical test that I can use to prove that the proposed ml is better or not? Thanks.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

I did something like that when I was at university. Professor asked me to implement and compare a variant of k-nearest neighbor with the original one.

I've applied the 10 fold cross validation over several datasets with both the algorithms and then I've applied the Friedman test with Holm post hoc over the accuracy means. You can use also simple paired t-test or others, it depends on your needs.

My professor also suggested me to use this graphical tool. Hope this helps

SPOILER

It turned out that my k-NN variant wasn't better!

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ The Friedman test is the non-parametric alternative to the one-way ANOVA with repeated measures. It is used to test for differences between groups when the dependent variable being measured is ordinal. Also, paired t-tests are also used for repeated measures. $\endgroup$
    – user2974951
    Sep 19, 2018 at 6:49
0
$\begingroup$

Tests that are used to compare models include ANOVA (Chi-square based tests, F-tests), log likelihood based tests (deviance, Wilk's lambda) or AIC / BIC based tests (penalized deviance).

Improve this answer
$\endgroup$
0
$\begingroup$

Not sure that we can use parametric methods here, as we don't have information about distribution parameters.

Generally, I agree with @ggagliano. Sharing my own experience, I used non-parametric sign-test for paired samples.

For example we have the following results for C4.5 classificator with parameter tuning.

enter image description here

As a statistics for this method we use the sum of indicators where the first classificator is better than the second one. If the first hypothesis is valid, then we have a binomial distribution with parameters (n,1/2). Which means that both classificators are equal. The alternative hypothesis claims that the quality of modified classificator is better.

The p-value for this example is 0.019. On a significance level of 0.05 we can reject the first hypothesis.

You can try to use the same approach in your work as well.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.