# Advice and Ideas appreciated - Machine Learning one man project

I have a project where I am supposed to start from scratch and learn how machine Learning works. So far everything is working out better than expected but I feel as I am offered to many ways to choose from.

My Project:
I have data with 700 rows and 108 columns as my features and I get pretty decent results when using a RandomForestClassifier.
By now I was using train_test_split to split my data but I was reading a lot of articles where it was recommended to split data into 3 Sets (train, dev, test).
Since I dont have that much Data I thought of using a Cross-Validation.

My Problem:
So I implemented it, but couldn't really find the difference between a CV and a train_test_split with shuffle.
Before doing so I thought I know the difference betweeen these to model-selection strategies but now I am a bit confused.

My Knowledge And Questions:
1. train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples.
--> can't that be solved by using stratify=True?
2. train_test_split doesn't split the sets the same way all the time so the results aren't comparable
--> setting random_state=0 solves the problem?
3. When does it make sence for me to use CV?
4. How to understand the following picture and what benefit does it provide:

1. What would be a good way to proceed?

thanks a lot for all the help and time in advance! Cheers!

1. train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples. --> can't that be solved by using stratify=True? --> yes, that's what stratify=True is for. However you still only train on the data of your training set and test with the data from the test set

2. train_test_split doesn't split the sets the same way all the time so the results aren't comparable --> setting random_state=0 solves the problem? --> it does...keep in mind that random_state= will work fine as long as you keep the seed aka random-state the same

for 3. and 4. let me make sure you understand the difference between CV and TTS. TTS splits your data once, trains on the now "fixed" training set and tests on the "fixed" test set. However this introduces a kind of bias to your evaluation because you are not training or testing on all observations. By going for a CV you make sure that all observations are used for testing and training. This reduces the bias of setting a "fixed" training and test set.

Now for 3.: When in doubt use CV. HOWEVER...sometimes this doesn't make sense. Think about time series data and how cv would work then and what the result would be. That being said...now think about how TTS with stratify or random would work and what those would mean. CV usually is a good way to go however you always have to think about if the evaluation method fits your problem. TTS has its merit for time series as well as for performance reasons using large datasets.

And 4.: The picture visualizes TTS and CV. First you have CV which splits your training data so that each small data set is used as test data. The "Testing" set is then used to measure generalization of your model. This is called the holdout method

The diagram that says "Data Permitting" demonstrates TTS with an additional "Testing" set for measuring generalization. You train on "Training", validate the model on "Validation" and as with the holdout method you keep a "Testing" set for measuring how well your model generalizes on unknown data.

Last but not least...5.: If you get good results and generalization by using TTS...go for it. If you get problems with over-/underfitting or generalization try CV and see if it helps. Of course as with everything in Data Science and Machine Learning there are multiple levels of complexity that you can add on top of TTS and CV as well. For example you could read about K-Fold or LOOCV etc. I don't think that is necessary though.

• Very helpful, as I am not having that much Data, CV is the way to go for? I thought of using TTS and using cross_val_score on it. When doing so, I dont have to export the final model and use it on the whole data again right? atleast if I understand the documentation of cross_val_scoreright, then it should be so. I allready read about KFold and LOOCV but I don't see the use for them unless the data requires a certain amount of folds to be used. If you could elaborate on this, It would be great! Oct 2, 2019 at 14:22
• I think cv doesn't hurt with your amount of data (as long as you don't have time series related data then you'll have to be a bit careful with that). Holding out a part of your data set for a generalization check would be advised in most use cases as well. cross_val_score is fine. By default it will do a KFold cross validation anyway. The higher you set your k the smaller your testing set will be. Take this as a hyperparameter to play around with for the start. I don't think you have to bother with LOOCV for now. The tl;dr version: You do a K fold where k is the number of observations Oct 2, 2019 at 14:46

1. Indeed, if you aren't careful when creating dataset partitions, you can end up with unbalanced partitions. In that case, as you said, using stratified sampling can mitigate the problem. Another option is to perform data augmentation on feature space (i.e. SMOTE [1]).

2. For comparable results, you should use a seed/random_state. That depends on your implementation (for numpy, for instance, it is common to use np.random.seed(0)).

3. That is an important question. The purpose of the Test set is to see how your model generalizes on unseen data (as models are normally optimistic on training data), so that you can detect, for instance, overfitting problems. The validation set, however, has another purpose: when tuning hyperparameters, it is recommended to use another partition not present on train, nor test datasets. With that in mind, unless you are tuning your parameters, the Valid partition is equivalent to Test partition.

4. The cross-validation strategy is used for model selection/evaluation. In that scheme, you make N partitions in your entire dataset (Train + Test), say $$\{\mathcal{T}_{j}\}_{j=1}^{N}$$. Your model will be trained on N - 1 partitions while leaving a specific partition, say $$\mathcal{T}_{i}$$, for testing. You repeat this process for each $$i=1,\cdots,N$$. In the evaluation end, you will get $$N$$ estimates of your metrics (i.e. accuracy), so that you can estimate the expected value for them, and your confidence in that value.

5. I would recommend you to adopt stratified sampling, seeding your pseudo-random number generator and don't use validation data, unless you are tuning hyperparameters for your model.

References

[1] Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, 321-357.

• can you try to explain point 3 with other words, im uncertain whether I grasped what you have been trying to say. Espacially the last part. (isn't the validation set like the 'k'-fold in KFoldCV, so the left out part of the original training set to validate the model with - as shown in the picture?) And what do you mean with your last sentence? (Valid Partition) To the 5th point stratified sampling means TTS with stratify? And why is it important to Validate when tuning HP but later on it isn't? (my guess: for beeing sure the model is not biased) Oct 2, 2019 at 14:34
• Well, I don't see the validation set/partition as kFoldCV. The idea of kFoldCV is to get a deeper glimpse into how your model is performing on average, while the validation set is used for HP tuning. I remark that you need to separate your data for tuning to get comparable metrics. Oct 2, 2019 at 14:55
• Moreover, I mean by stratified sampling (en.wikipedia.org/wiki/Stratified_sampling): getting your samples in a class-wise fashion, instead of taking them from the entire population. Say, if you want 20 samples, you take 10 from Class A (at random), and 10 from class B, and not 20 at random from classes A and B. Oct 2, 2019 at 14:57