1
$\begingroup$

I am using one file for training (e.g train.arff) and another for testing (e.g test.atff) with the 70-30 ratio in Weka. I want to ask how can I use the repeated training/testing in Weka when I have separate train and test data files and the second part of the question is what is the advantage if we use repeated and what if we dont use it? Thanks in advance

$\endgroup$
1
$\begingroup$

In general the advantage of repeated training/testing is to measure to what extent the performance is due to chance. The most common source of chance comes from which instances are selected as training/testing data. One can use k-fold cross-validation in order to mitigate the effect of chance in this case. Weka performs 10-fold CV by default, as far as I remember, but this is not compatible with providing a specific training/test set.


[edit based on OP's comments]

In the video mentioned by OP, the author loads a dataset and sets the "percentage split" at 90%. This means that the full dataset will be split between training and test set by Weka itself. Weka randomly selects which instances are used for training, this is why chance is involved in the process and this is why the author proceeds to repeat the experiment with different values for the random seed: every time Weka will selects a different subset of instances as training set, resulting in a different accuracy. In other words, the purpose of repeating the experiment is to change how the dataset is split between training and test set. In this case (J48 with default options) there would be no point repeating the experiment with a fixed training set, because there's no chance involved in the process so there's no variation in the result.

It's worth noticing that this lesson by the author of the video seems to be used as an introduction to the more general concept of k-fold cross-validation, presented a couple of lessons later in the course.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ can we use the repeated train/test when we provide a separate test set, or just we can do it using k-fold CV and percentage split? $\endgroup$ – Javed Khan Jul 20 '19 at 16:06
  • $\begingroup$ The problem is that cross-validation works by changing the split between training and test set, so it's not compatible with a single test set. however it's possible to perform CV yourself and provide a different pair of training/test set to Weka repeatedly. You'll find a lot of explanations about cross-validation on stats.stackexchange.com (it's called cross-validated so it's quite appropriate!), for instance this one stats.stackexchange.com/questions/148688/… but check also the related questions to understand how CV works. $\endgroup$ – Erwan Jul 20 '19 at 16:33
  • $\begingroup$ In general repeating the exact same training stage with the same training data wouldn't be very useful (unless the training method strongly depends on some random seed, but I don't think that's your case) $\endgroup$ – Erwan Jul 20 '19 at 16:37
  • $\begingroup$ But I was watching a video from Ian (from Weka team) and he applied on the same training set with J48 model. ? $\endgroup$ – Javed Khan Jul 20 '19 at 19:12
  • $\begingroup$ With Percentage split set to 90% for different random seed values he got different results. $\endgroup$ – Javed Khan Jul 20 '19 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.