# Linear Regression and k-fold cross validation

I am totally new to the topic of Data Science. With the help of the following sources, I think I have managed to do a very simple and basic Linear regression on a train dataset:

My Python code (written as an iPython notebook) that actually does the computation looks like this:

### Stage 0: "Import some stuff"
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
from sklearn.linear_model import LinearRegression

### Stage 1: "Prepare train dataset"

### remove categorical cols
only_numerical_train_dataset = my_train_dataset.loc[:, my_train_dataset.dtypes!=object]

### remove 'Id' and 'SalePrice' columns
my_train_dataset_X = only_numerical_train_dataset.drop(['Id','SalePrice'], axis = 1)

### insert median into cells with missing values
print("Before: Number of cells with missing values in train data: " + str(np.sum(np.sum(my_train_dataset_X.isnull()))))
null_values_per_col = np.sum(my_train_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("Before: Need to replace values in the columns in train data: " + str(cols_to_impute) + "\n")
imputation_val_for_na_cols = dict()
for col in cols_to_impute:
if (my_train_dataset_X[col].dtype == 'float64' ) or  (my_train_dataset_X[col].dtype == 'int64'):
#numerical col
imputation_val_for_na_cols[col] = np.nanmedian(my_train_dataset_X[col]) #with median
for key, val in imputation_val_for_na_cols.items():
my_train_dataset_X[key].fillna(value= val, inplace = True)
print("After: Number of cells with missing values in train data: " + str(np.sum(np.sum(my_train_dataset_X.isnull()))))
null_values_per_col = np.sum(my_train_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("After: Need to replace values in the columns in train data: " + str(cols_to_impute) + "\n")

### Stage 2: "Sanity Check - the better the quality, the higher the price?"
plt.scatter(my_train_dataset.OverallQual, my_train_dataset.SalePrice)
plt.xlabel("Overall Quality of the house")
plt.ylabel("Price of the house")
plt.title("Relationship between Price and Quality")
plt.show()

### Stage 3: "Prepare the test dataset"

### remove categorical cols
only_numerical_test_dataset = my_test_dataset.loc[:, my_test_dataset.dtypes!=object]

### remove 'Id' column
my_test_dataset_X = only_numerical_test_dataset.drop(['Id'], axis = 1)

### insert median into cells with missing values
print("Before: Number of cells with missing values in test data: " + str(np.sum(np.sum(my_test_dataset_X.isnull()))))
null_values_per_col = np.sum(my_test_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("Before: Need to replace values in the columns in test data: " + str(cols_to_impute) + "\n")
imputation_val_for_na_cols = dict()
for col in cols_to_impute:
if (my_test_dataset_X[col].dtype == 'float64' ) or  (my_test_dataset_X[col].dtype == 'int64'):
#numerical col
imputation_val_for_na_cols[col] = np.nanmedian(my_test_dataset_X[col]) #with median
for key, val in imputation_val_for_na_cols.items():
my_test_dataset_X[key].fillna(value= val, inplace = True)
print("After: Number of cells with missing values in test data: " + str(np.sum(np.sum(my_test_dataset_X.isnull()))))
null_values_per_col = np.sum(my_test_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("After: Need to replace values in the columns in test data: " + str(cols_to_impute) + "\n")

### Stage 4: "Apply the model"
lm = LinearRegression()
lm.fit(my_train_dataset_X, my_train_dataset.SalePrice)

### Stage 5: "Sanity Check - the better the quality, the higher the predicted SalesPrice?"
plt.scatter(my_test_dataset.OverallQual, lm.predict(my_test_dataset_X))
plt.xlabel("Overall Quality of the house in test data")
plt.ylabel("Price of the house in test data")
plt.title("Relationship between Price and Quality in test data")
plt.show()

### Stage 6: "Check the performance of the Prediction"
from sklearn.model_selection import cross_val_score
scores = cross_val_score(lm, my_train_dataset_X,  lm.predict(my_test_dataset_X), cv=10)
print("scores = " + str(scores))


## My questions are:

1. Why am I getting an error in Stage 6 and how to fix it?

ValueError Traceback (most recent call last)
<ipython-input-2-700c31f0d410> in <module>()
85 ### test the performance of the model
86 from sklearn.model_selection import cross_val_score
---> 87 scores = cross_val_score(lm, my_train_dataset_X,  lm.predict(my_test_dataset_X), cv=10)
88 print("scores = " + str(scores))
89
ValueError: Found input variables with inconsistent numbers of samples: [1460, 1459]


2. Is there something fundamentally wrong with my approach to a simple and basic Linear Regression?

@CalZ - First comment:

my_test_dataset_X.shape = (1459, 36)
my_train_dataset_X.shape = (1460, 36)


@CalZ - Second comment: I will consider refactoring the code as soon as I am sure that my approach is not fundamentally wrong.

• It thinks two objects are not comparable. Run these and see what they say: my_test_dataset_X.shape, my_train_dataset_X.shape. – CalZ Jun 9 '17 at 12:43
• Also, you could save a lot code by loading your train and test together into one dataframe and doing imputation once. Alternatively, put the imputation into a function so you aren't writing the same code twice. – CalZ Jun 9 '17 at 12:44
• @CalZ Technically, you should not mix the training and test sets for imputation, or any such operation (the reason being leakage of information from the training set into the test set). I'm not sure though how much this will influence the end result. – darXider Jun 9 '17 at 13:40

1. As the error message states, the invocation to cross_val_score fails because the shape arguments differ in their first dimension (1460 vs. 1459). This is consistent with the number of lines in the CSV files. However, the underlying problem is that you are mixing the test and the training sets. You should invoke it only with the test set: cross_val_score(lm, my_test_dataset_X, lm.predict(my_test_dataset_X), cv=10). Update: My initial suggestion was NOT correct, you cannot use your own predictions to validate! You should leave a subset of the labeled data for hold out on which to compute the cross validation.
• regarding 1.: I changed it and i get these scores: scores = [ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] . Does it mean my model is extremely accurate or totally inaccurate? – kiltek Jun 9 '17 at 13:13