# Split the data between the Training Data and Test Data using sklearn

Work to do

My job is to take the data and divide it between Training and Test using 30% of the data as Test where both should have the same ratio between positive and negative.

CSV File

age,Feature 2,Feature 3,Feature 4,income,Feature 6,Feature 7,Feature 8,Feature 9,Feature 10,Feature 11,Feature 12,Feature 13,Feature 14,Feature 15,Class
77,1,0,0,3,0,1,1,1,0,1,1,1,0,1,0
35,1,0,1,4,0,0,0,1,1,0,1,1,0,1,0
79,1,0,0,2,0,1,1,0,0,0,1,1,1,0,1
61,0,1,0,1,0,1,1,1,0,0,0,0,1,0,0
62,0,0,0,2,0,1,0,0,0,1,1,0,0,0,1
63,0,1,1,0,1,1,0,0,1,0,0,1,0,0,1
29,1,1,0,1,1,0,1,1,0,0,1,1,1,1,0
39,0,1,1,5,1,1,1,0,1,0,1,1,1,1,0
51,0,1,1,6,1,1,0,1,0,0,1,1,0,1,1



Code - Training Data and Test Data

#!/usr/bin/env python3

import numpy as np
import pandas as pd
import matplotlib.pyplot as plot
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import Imputer

X = dataSet.iloc[:,:-1].values
y = dataSet.iloc[:,15].values

df = pd.DataFrame(dataSet)
print(df)

# Missing data
imputer = Imputer(missing_values="NaN" , strategy="mean" , axis=0)

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y , test_size = 0.30 , random_state = 0)

# Creating linear regression object
linearReg = LinearRegression()
linearReg.fit(xTrain , yTrain)

#Now, testing model
yPrediction = linearReg.predict(xTest)

print(X.shape)
print(y.shape)
print(xTrain , yTrain)

#Plotting the training set
plot.scatter(xTrain , yTrain , color="red")
plot.plot(xTrain , linearReg.predict(xTrain) , color="blue")
plot.title("title")
plot.xlabel("x label")
plot.ylabel("y label")
plot.show()

# Test set
plot.scatter(xTest, yTest, color = 'red')
plot.plot(xTrain, linearReg.predict(xTrain), color = 'blue')
plot.title("title")
plot.xlabel("x label")
plot.ylabel("y label")
plot.show()



Problems found

The following error is showing me: ValueError: x and y must be the same size

This is because apparently in the part of

X = dataSet.iloc [:,: - 1] .values
y = dataSet.iloc [:, 15] .values


I'm not taking the correct values. If someone can help me to correct the error I will thank you

After addingstratify = y to:

xTrain , xTest , yTrain , yTest = train_test_split(X , y , test_size = 0.30, random_state = 0 , stratify = y)


It shows me the following graph, in which I presume that it is incorrect

What you are looking for is called Stratified sampling

From this CrossValidated question, we have a short explanation

Stratified sampling aims at splitting one data set so that each split are similar with respect to something. In a classification setting, it is often chosen to ensure that the train and test sets have approximately the same percentage of samples of each target class as the complete set.

To get a stratified split in ScikitLearn, you just need to edit this part of your code

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y ,
test_size = 0.30 ,
random_state = 0,
----->  stratify = y)


This will automatically split your dataset in train and test but also keep the same proportion of positives and negatives as the original dataset.

So, if you have a dataset where the positive class is 70% of the records and negative class 30%, after a stratified split, both train and test will have the same 70-30 distribution.

• Thank you very much for the information, very valuable. My only problem is that when the graphics are generated it shows me many lines in different directions. May 2, 2019 at 15:03
• I just did update, I added the graph. May 2, 2019 at 15:12

I would like to add here that it is important to remember the Blind Test Rule. In particular, in order to avoid accidentally optimising your code for the test data during development, a golden rule is to split the data set in three datasets:

1. Train dataset, to do the training (i.e. 60%)
2. Validation dataset, to test during development (i.e. 20%), and
3. Test dataset, to do the final testing only at the end of the development (i.e. 20%).

Just print type of both xTest and yTest. You should see that both are of different type. Or simply print both of them. You should be able to see that they are indeed different.

I think using xTest[:,0] for plotting should solve the problem.

The same thing applies for xTrain.

• Sorry, in which part of the code you recommend me to make the change for xTest [:, 0]? May 1, 2019 at 18:11
• you have to change both xTest and xTrain. The following should solve your problem: plot.scatter(xTrain[:,0], yTrain , color="red") May 2, 2019 at 1:40
• Thanks, I just fixed it. But looking at the graphics, I think the results are incorrect. May 2, 2019 at 1:51
• What I want to achieve is to take the data and divide it between Training and Test using 30% of the data as Test where both should have the same ratio between positive and negative. May 2, 2019 at 1:53
• use: scikit-learn.org/stable/modules/generated/… Or train_test_split(X , y , test_size = 0.30 , random_state = 0, stratify=y) May 2, 2019 at 1:58