1
$\begingroup$

I am using one file for training (e.g train.arff) and another for testing (e.g test.atff) with the 70-30 ratio in Weka. I want to ask how can I use the repeated training/testing in Weka when I have separate train and test data files and the second part of the question is what is the advantage if we use repeated and what if we dont use it? Thanks in advance

$\endgroup$

1 Answer 1

1
$\begingroup$

In general the advantage of repeated training/testing is to measure to what extent the performance is due to chance. The most common source of chance comes from which instances are selected as training/testing data. One can use k-fold cross-validation in order to mitigate the effect of chance in this case. Weka performs 10-fold CV by default, as far as I remember, but this is not compatible with providing a specific training/test set.


[edit based on OP's comments]

In the video mentioned by OP, the author loads a dataset and sets the "percentage split" at 90%. This means that the full dataset will be split between training and test set by Weka itself. Weka randomly selects which instances are used for training, this is why chance is involved in the process and this is why the author proceeds to repeat the experiment with different values for the random seed: every time Weka will selects a different subset of instances as training set, resulting in a different accuracy. In other words, the purpose of repeating the experiment is to change how the dataset is split between training and test set. In this case (J48 with default options) there would be no point repeating the experiment with a fixed training set, because there's no chance involved in the process so there's no variation in the result.

It's worth noticing that this lesson by the author of the video seems to be used as an introduction to the more general concept of k-fold cross-validation, presented a couple of lessons later in the course.

$\endgroup$
10
  • $\begingroup$ can we use the repeated train/test when we provide a separate test set, or just we can do it using k-fold CV and percentage split? $\endgroup$
    – Khan
    Jul 20, 2019 at 16:06
  • $\begingroup$ The problem is that cross-validation works by changing the split between training and test set, so it's not compatible with a single test set. however it's possible to perform CV yourself and provide a different pair of training/test set to Weka repeatedly. You'll find a lot of explanations about cross-validation on stats.stackexchange.com (it's called cross-validated so it's quite appropriate!), for instance this one stats.stackexchange.com/questions/148688/… but check also the related questions to understand how CV works. $\endgroup$
    – Erwan
    Jul 20, 2019 at 16:33
  • $\begingroup$ In general repeating the exact same training stage with the same training data wouldn't be very useful (unless the training method strongly depends on some random seed, but I don't think that's your case) $\endgroup$
    – Erwan
    Jul 20, 2019 at 16:37
  • $\begingroup$ But I was watching a video from Ian (from Weka team) and he applied on the same training set with J48 model. ? $\endgroup$
    – Khan
    Jul 20, 2019 at 19:12
  • 1
    $\begingroup$ Yes, exactly. CV consists in using the same dataset for repeated experiments which differ by changing the instances as training set. There are also other similar techniques (such as bagging: en.wikipedia.org/wiki/Bootstrap_aggregating), but CV is the most convenient and consequently the most standard. $\endgroup$
    – Erwan
    Jul 22, 2019 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.