Given a single discrete data set, how should I divide it into training data and test data?

Question

I have a dataset in libSVM format consisting of 6000 entries, each with 5 indices, and each index has a binary value 1 or 2. Each of the 6000 entries has a label of 1 or 0, and I am trying to use various machine learning algorithms to determine the correct label (0 or 1) given a particular set of 5 indices/values.

For example, consider the following dataset (the real one is 6000 lines):

0 101:1 102:1 103:0 104:1 105:1
0 101:0 102:1 103:0 104:1 105:1
0 101:0 102:1 103:1 104:1 105:1
1 101:1 102:1 103:1 104:1 105:1
1 101:0 102:1 103:0 104:0 105:1
1 101:1 102:1 103:1 104:0 105:0
1 101:0 102:1 103:0 104:0 105:0

For an algorithm that predicts binary classification, like xgboost, conceptually, how do I first use my dataset to train the model, and then apply the model to the data?

I ask because xgboost asks for two files, a data training set and a data test set. It seems to me that the algorithm should just require a single full set of data, use all of the data to train and build a model, and then apply that model to the original data set and determine if the labels are being assigned "0 or 1" accurately.

Any help in understanding this concept is much appreciated.

Answer 1

In machine learning, it is important to test out the model that you have built on your training data. This is to prevent overfitting. This is why you must split your data into testing and training. There are many different ways to split testing and training. You can randomly split the data set so that 80% of the samples are training and 20% are testing. Something else you may want to consider is using stratified sampling so that the positive labels occur in both testing and training. This is especially important if you only have a few positively labeled samples, as you could easily end up with a test set without any positive samples. In python there is an argument ‘stratify’ that you can use so that the split has balanced classes.

Answer 2

Assuming you are using Python, an easy way to do this is to use utilities available in scikit-learn:

from sklearn.datasets import load_svmlight_file, dump_svmlight_file
from sklearn.model_selection import train_test_split

# load features and labels
X, y = load_svmlight_file('path/to/libsvm/data')

# split into train/test sets (change test_size if you like)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# write the train & test datasets to disk
dump_svmlight_file(X_train, y_train, 'train.svm')
dump_svmlight_file(X_train, y_train, 'test.svm')

In reference to your comment

It seems to me that the algorithm should just require a single full set of data, use all of the data to train and build a model, and then apply that model to the original data set and determine if the labels are being assigned "0 or 1" accurately.

I would recommend reading about overfitting. In short, overfitting happens if your model is very good at classifying the data that you used to train the model, but performs poorly on unseen data. If you fit a model to a dataset, and then test the model on the same dataset, you will likely get very optimistic estimates for performance that may lead you to believe that your model is much better than it actually is.

After finding a set of hyper-parameters that work well and testing to ensure that your model isn't overfitting, you can train the model on the full dataset using the hyper-parameters that worked.

Some good references on overfitting:

• Why Is Overfitting Bad in Machine Learning?
• Overfitting in Machine Learning: What It Is and How to Prevent It