0
$\begingroup$

I have a dataset in libSVM format consisting of 6000 entries, each with 5 indices, and each index has a binary value 1 or 2. Each of the 6000 entries has a label of 1 or 0, and I am trying to use various machine learning algorithms to determine the correct label (0 or 1) given a particular set of 5 indices/values.

For example, consider the following dataset (the real one is 6000 lines):

0 101:1 102:1 103:0 104:1 105:1
0 101:0 102:1 103:0 104:1 105:1
0 101:0 102:1 103:1 104:1 105:1
1 101:1 102:1 103:1 104:1 105:1
1 101:0 102:1 103:0 104:0 105:1
1 101:1 102:1 103:1 104:0 105:0
1 101:0 102:1 103:0 104:0 105:0

For an algorithm that predicts binary classification, like xgboost, conceptually, how do I first use my dataset to train the model, and then apply the model to the data?

I ask because xgboost asks for two files, a data training set and a data test set. It seems to me that the algorithm should just require a single full set of data, use all of the data to train and build a model, and then apply that model to the original data set and determine if the labels are being assigned "0 or 1" accurately.

Any help in understanding this concept is much appreciated.

$\endgroup$
2
$\begingroup$

In machine learning, it is important to test out the model that you have built on your training data. This is to prevent overfitting. This is why you must split your data into testing and training. There are many different ways to split testing and training. You can randomly split the data set so that 80% of the samples are training and 20% are testing. Something else you may want to consider is using stratified sampling so that the positive labels occur in both testing and training. This is especially important if you only have a few positively labeled samples, as you could easily end up with a test set without any positive samples. In python there is an argument ‘stratify’ that you can use so that the split has balanced classes.

$\endgroup$
  • $\begingroup$ You can also use crossvalidation: Split your data into $k$ parts, train with $k-1$ and test with 1, repeat until all parts have been used for testing. This is called $k$-fold cross validation and will tell you how trustable is your model (by the variance in results) and how precise (by the mean result). Also you can do leave-on-out crossvalidation which is the same as a $k$-fold cv where $k$ equals to the size of dataset. $\endgroup$ – Pedro Henrique Monforte Apr 11 at 3:25
  • $\begingroup$ @PedroHenriqueMonforte I completely agree, cross validation is a better route than just using one test set. However -- since this individual seems to be new to ML and confused about the general concept, I decided to keep my response limited. $\endgroup$ – fractalnature Apr 11 at 16:15
  • $\begingroup$ I understood your approach, this is why i just added a comment for others looking for it to see and not edit your answer. Also I've upvoted your answer $\endgroup$ – Pedro Henrique Monforte Apr 11 at 16:17
0
$\begingroup$

Assuming you are using Python, an easy way to do this is to use utilities available in scikit-learn:

from sklearn.datasets import load_svmlight_file, dump_svmlight_file
from sklearn.model_selection import train_test_split

# load features and labels
X, y = load_svmlight_file('path/to/libsvm/data')

# split into train/test sets (change test_size if you like)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# write the train & test datasets to disk
dump_svmlight_file(X_train, y_train, 'train.svm')
dump_svmlight_file(X_train, y_train, 'test.svm')

In reference to your comment

It seems to me that the algorithm should just require a single full set of data, use all of the data to train and build a model, and then apply that model to the original data set and determine if the labels are being assigned "0 or 1" accurately.

I would recommend reading about overfitting. In short, overfitting happens if your model is very good at classifying the data that you used to train the model, but performs poorly on unseen data. If you fit a model to a dataset, and then test the model on the same dataset, you will likely get very optimistic estimates for performance that may lead you to believe that your model is much better than it actually is.

After finding a set of hyper-parameters that work well and testing to ensure that your model isn't overfitting, you can train the model on the full dataset using the hyper-parameters that worked.

Some good references on overfitting:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.