Advice and Ideas appreciated - Machine Learning one man project

Question

I have a project where I am supposed to start from scratch and learn how machine Learning works. So far everything is working out better than expected but I feel as I am offered to many ways to choose from.

My Project:
I have data with 700 rows and 108 columns as my features and I get pretty decent results when using a RandomForestClassifier.
By now I was using train_test_split to split my data but I was reading a lot of articles where it was recommended to split data into 3 Sets (train, dev, test).
Since I dont have that much Data I thought of using a Cross-Validation.

My Problem:
So I implemented it, but couldn't really find the difference between a CV and a train_test_split with shuffle.
Before doing so I thought I know the difference betweeen these to model-selection strategies but now I am a bit confused.

My Knowledge And Questions:
1. train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples.
--> can't that be solved by using stratify=True?
2. train_test_split doesn't split the sets the same way all the time so the results aren't comparable
--> setting random_state=0 solves the problem?
3. When does it make sence for me to use CV?
4. How to understand the following picture and what benefit does it provide:

5. What would be a good way to proceed?

thanks a lot for all the help and time in advance! Cheers!

Answer 1

1. train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples. --> can't that be solved by using stratify=True? --> yes, that's what stratify=True is for. However you still only train on the data of your training set and test with the data from the test set

2. train_test_split doesn't split the sets the same way all the time so the results aren't comparable --> setting random_state=0 solves the problem? --> it does...keep in mind that random_state= will work fine as long as you keep the seed aka random-state the same

for 3. and 4. let me make sure you understand the difference between CV and TTS. TTS splits your data once, trains on the now "fixed" training set and tests on the "fixed" test set. However this introduces a kind of bias to your evaluation because you are not training or testing on all observations. By going for a CV you make sure that all observations are used for testing and training. This reduces the bias of setting a "fixed" training and test set.

Now for 3.: When in doubt use CV. HOWEVER...sometimes this doesn't make sense. Think about time series data and how cv would work then and what the result would be. That being said...now think about how TTS with stratify or random would work and what those would mean. CV usually is a good way to go however you always have to think about if the evaluation method fits your problem. TTS has its merit for time series as well as for performance reasons using large datasets.

And 4.: The picture visualizes TTS and CV. First you have CV which splits your training data so that each small data set is used as test data. The "Testing" set is then used to measure generalization of your model. This is called the holdout method

The diagram that says "Data Permitting" demonstrates TTS with an additional "Testing" set for measuring generalization. You train on "Training", validate the model on "Validation" and as with the holdout method you keep a "Testing" set for measuring how well your model generalizes on unknown data.

Last but not least...5.: If you get good results and generalization by using TTS...go for it. If you get problems with over-/underfitting or generalization try CV and see if it helps. Of course as with everything in Data Science and Machine Learning there are multiple levels of complexity that you can add on top of TTS and CV as well. For example you could read about K-Fold or LOOCV etc. I don't think that is necessary though.

Answer 2

Answering the questions,

1. Indeed, if you aren't careful when creating dataset partitions, you can end up with unbalanced partitions. In that case, as you said, using stratified sampling can mitigate the problem. Another option is to perform data augmentation on feature space (i.e. SMOTE [1]).

2. For comparable results, you should use a seed/random_state. That depends on your implementation (for numpy, for instance, it is common to use np.random.seed(0)).

3. That is an important question. The purpose of the Test set is to see how your model generalizes on unseen data (as models are normally optimistic on training data), so that you can detect, for instance, overfitting problems. The validation set, however, has another purpose: when tuning hyperparameters, it is recommended to use another partition not present on train, nor test datasets. With that in mind, unless you are tuning your parameters, the Valid partition is equivalent to Test partition.

4. The cross-validation strategy is used for model selection/evaluation. In that scheme, you make N partitions in your entire dataset (Train + Test), say $\{\mathcal{T}_{j}\}_{j=1}^{N}$. Your model will be trained on N - 1 partitions while leaving a specific partition, say $\mathcal{T}_{i}$, for testing. You repeat this process for each $i=1,\cdots,N$. In the evaluation end, you will get $N$ estimates of your metrics (i.e. accuracy), so that you can estimate the expected value for them, and your confidence in that value.

5. I would recommend you to adopt stratified sampling, seeding your pseudo-random number generator and don't use validation data, unless you are tuning hyperparameters for your model.

References

[1] Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, 321-357.