# Linear Regression and k-fold cross validation

I am totally new to the topic of Data Science. With the help of the following sources, I think I have managed to do a very simple and basic Linear regression on a train dataset:

My Python code (written as an iPython notebook) that actually does the computation looks like this:

### Stage 0: "Import some stuff"
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
from sklearn.linear_model import LinearRegression

### Stage 1: "Prepare train dataset"

### remove categorical cols
only_numerical_train_dataset = my_train_dataset.loc[:, my_train_dataset.dtypes!=object]

### remove 'Id' and 'SalePrice' columns
my_train_dataset_X = only_numerical_train_dataset.drop(['Id','SalePrice'], axis = 1)

### insert median into cells with missing values
print("Before: Number of cells with missing values in train data: " + str(np.sum(np.sum(my_train_dataset_X.isnull()))))
null_values_per_col = np.sum(my_train_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("Before: Need to replace values in the columns in train data: " + str(cols_to_impute) + "\n")
imputation_val_for_na_cols = dict()
for col in cols_to_impute:
if (my_train_dataset_X[col].dtype == 'float64' ) or  (my_train_dataset_X[col].dtype == 'int64'):
#numerical col
imputation_val_for_na_cols[col] = np.nanmedian(my_train_dataset_X[col]) #with median
for key, val in imputation_val_for_na_cols.items():
my_train_dataset_X[key].fillna(value= val, inplace = True)
print("After: Number of cells with missing values in train data: " + str(np.sum(np.sum(my_train_dataset_X.isnull()))))
null_values_per_col = np.sum(my_train_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("After: Need to replace values in the columns in train data: " + str(cols_to_impute) + "\n")

### Stage 2: "Sanity Check - the better the quality, the higher the price?"
plt.scatter(my_train_dataset.OverallQual, my_train_dataset.SalePrice)
plt.xlabel("Overall Quality of the house")
plt.ylabel("Price of the house")
plt.title("Relationship between Price and Quality")
plt.show()

### Stage 3: "Prepare the test dataset"

### remove categorical cols
only_numerical_test_dataset = my_test_dataset.loc[:, my_test_dataset.dtypes!=object]

### remove 'Id' column
my_test_dataset_X = only_numerical_test_dataset.drop(['Id'], axis = 1)

### insert median into cells with missing values
print("Before: Number of cells with missing values in test data: " + str(np.sum(np.sum(my_test_dataset_X.isnull()))))
null_values_per_col = np.sum(my_test_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("Before: Need to replace values in the columns in test data: " + str(cols_to_impute) + "\n")
imputation_val_for_na_cols = dict()
for col in cols_to_impute:
if (my_test_dataset_X[col].dtype == 'float64' ) or  (my_test_dataset_X[col].dtype == 'int64'):
#numerical col
imputation_val_for_na_cols[col] = np.nanmedian(my_test_dataset_X[col]) #with median
for key, val in imputation_val_for_na_cols.items():
my_test_dataset_X[key].fillna(value= val, inplace = True)
print("After: Number of cells with missing values in test data: " + str(np.sum(np.sum(my_test_dataset_X.isnull()))))
null_values_per_col = np.sum(my_test_dataset_X.isnull(), axis=0)
cols_to_impute = []
for key in null_values_per_col.keys():
if null_values_per_col.get(key) != 0:
cols_to_impute.append(key)
print("After: Need to replace values in the columns in test data: " + str(cols_to_impute) + "\n")

### Stage 4: "Apply the model"
lm = LinearRegression()
lm.fit(my_train_dataset_X, my_train_dataset.SalePrice)

### Stage 5: "Sanity Check - the better the quality, the higher the predicted SalesPrice?"
plt.scatter(my_test_dataset.OverallQual, lm.predict(my_test_dataset_X))
plt.xlabel("Overall Quality of the house in test data")
plt.ylabel("Price of the house in test data")
plt.title("Relationship between Price and Quality in test data")
plt.show()

### Stage 6: "Check the performance of the Prediction"
from sklearn.model_selection import cross_val_score
scores = cross_val_score(lm, my_train_dataset_X,  lm.predict(my_test_dataset_X), cv=10)
print("scores = " + str(scores))


## My questions are:

1. Why am I getting an error in Stage 6 and how to fix it?

ValueError Traceback (most recent call last)
<ipython-input-2-700c31f0d410> in <module>()
85 ### test the performance of the model
86 from sklearn.model_selection import cross_val_score
---> 87 scores = cross_val_score(lm, my_train_dataset_X,  lm.predict(my_test_dataset_X), cv=10)
88 print("scores = " + str(scores))
89
ValueError: Found input variables with inconsistent numbers of samples: [1460, 1459]


2. Is there something fundamentally wrong with my approach to a simple and basic Linear Regression?

@CalZ - First comment:

my_test_dataset_X.shape = (1459, 36)
my_train_dataset_X.shape = (1460, 36)


@CalZ - Second comment: I will consider refactoring the code as soon as I am sure that my approach is not fundamentally wrong.

• It thinks two objects are not comparable. Run these and see what they say: my_test_dataset_X.shape, my_train_dataset_X.shape.
– CalZ
Jun 9, 2017 at 12:43
• Also, you could save a lot code by loading your train and test together into one dataframe and doing imputation once. Alternatively, put the imputation into a function so you aren't writing the same code twice.
– CalZ
Jun 9, 2017 at 12:44
• @CalZ Technically, you should not mix the training and test sets for imputation, or any such operation (the reason being leakage of information from the training set into the test set). I'm not sure though how much this will influence the end result. Jun 9, 2017 at 13:40

1. As the error message states, the invocation to cross_val_score fails because the shape arguments differ in their first dimension (1460 vs. 1459). This is consistent with the number of lines in the CSV files. However, the underlying problem is that you are mixing the test and the training sets. You should invoke it only with the test set: cross_val_score(lm, my_test_dataset_X, lm.predict(my_test_dataset_X), cv=10). Update: My initial suggestion was NOT correct, you cannot use your own predictions to validate! You should leave a subset of the labeled data for hold out on which to compute the cross validation.

2. Yours is not only a linear regression. The bulk of your code is in charge of data manipulation (feature selection, data imputation) and not linear regression. Actually, you are reusing scikit-learn's implementation of linear regresion, not coding your own. If you want a code review of your snippet, maybe you should try in http://codereview.stackexchange.com (I don't know if this fits there either, you'd better check their help center).

UPDATE: About whether your code is sound from a data science point of view, it seems to me (after only a quick review) that you are doing reasonable things. There are some things that could be improved, like only handling float64 and int64 (while you can do as described here), only imputing NaNs and Nones (while there can be other values that should be imputed in certain cases, like outliers), or imputing blindly with the median (which is a safe decision but should be assessed taking into account the nature of each variable). But generally speaking seems Ok.

• regarding 2.: I am aware of the fact, that i am using a pre-implemented Linear regression and that i am doing feature selection and data imputation. I thought that my python comments would indicate that. In question 2. I want to know from the perspective of the data scientist wether or not it makes sense to engineer the data, the Sanity checks and the Linear regression like that. Jun 9, 2017 at 13:09
• regarding 1.: I changed it and i get these scores: scores = [ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] . Does it mean my model is extremely accurate or totally inaccurate? Jun 9, 2017 at 13:13
• It returns 1's because you are validating against your very own predictions. I have updated the answer to indicate that you cannot validate on an unlabeled dataset and that you should use a holdout dataset.
– noe
Jun 9, 2017 at 13:29
• How can I validate the model correctly? Jun 9, 2017 at 13:34
• As I suggested, you should take your labeled data (i.e. the train set) and split it in 2 parts. Then train your data with one of the parts (a typical ratio is to use 70% of your labeled data for training) and validate using the other part. Scikit learn has stuff for this.
– noe
Jun 9, 2017 at 13:38