# Logistic Regression doesn't predict for the entire test set

I am working through Kaggle's Titanic competition. I am mostly done with my model but the problem is that the logistic regression model does not predict for all of 418 rows in the test set but instead just returns predictions for 197 rows. This is the error PyCharm gives:

Traceback (most recent call last):
submission = pd.DataFrame({'PassengerId': test['PassengerId'], 'Survived': predictions})
File "C:\Users\security\Anaconda3\envs\TItanic-Kaggle.py\lib\site-packages\pandas\core\frame.py", line 392, in __init__
mgr = init_dict(data, index, columns, dtype=dtype)
File "C:\Users\security\Anaconda3\envs\TItanic-Kaggle.py\lib\site-packages\pandas\core\internals\construction.py", line 212, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "C:\Users\security\Anaconda3\envs\TItanic-Kaggle.py\lib\site-packages\pandas\core\internals\construction.py", line 51, in arrays_to_mgr
index = extract_index(arrays)
File "C:\Users\security\Anaconda3\envs\TItanic-Kaggle.py\lib\site-packages\pandas\core\internals\construction.py", line 328, in extract_index
raise ValueError(msg)
ValueError: array length 197 does not match index length 418


When I print(predictions) to confirm, this is what it gives:

[0 0 0 1 1 1 1 1 1 1 0 1 0 1 1 1 0 0 0 0 0 1 0 0 1 1 0 1 1 1 0 1 0 1 0 1 0
0 0 0 0 0 0 0 1 0 0 1 0 0 1 1 1 0 1 0 0 0 0 0 0 0 1 0 1 0 1 0 1 1 1 0 0 0
0 1 1 0 0 0 0 0 1 0 0 1 1 0 1 1 0 0 0 1 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 1 0
1 0 1 0 1 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 0 1 1 1 0 1
1 0 0 1 1 0 1 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 0 0
0 1 0 0 1 1 0 1 1 0 0 0]


This is my full code:

import pandas as pd
import warnings
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

warnings.filterwarnings("ignore", category=FutureWarning)

train['Sex'] = train['Sex'].replace(['female', 'male'], [0, 1])
train['Embarked'] = train['Embarked'].replace(['C', 'Q', 'S'], [1, 2, 3])

# Fill missing values in Age feature with each sex’s median value of Age
train['Age'].fillna(train.groupby('Sex')['Age'].transform("median"), inplace=True)

# Creating a new column called "HasCabin", where passengers with a cabin will get a score of 1 and those without cabins will get a score of 0
train['HasCabin'] = train['Cabin'].notnull().astype(int)

train['Relatives'] = train['SibSp'] + train['Parch']

logReg = LogisticRegression()

data = train[['Pclass', 'Sex', 'Relatives', 'Fare', 'Age', 'Embarked', 'HasCabin']]

# implement train_test_split
x_train, x_test, y_train, y_test = train_test_split(data, train['Survived'], test_size=0.22, random_state=0)

# Training the model with the Logistic Regression algorithm
logReg.fit(x_train, y_train)

predictions = logReg.predict(x_test)
submission = pd.DataFrame({'PassengerId': test['PassengerId'], 'Survived': predictions})

filename = 'Titanic-Submission.csv'
submission.to_csv(filename, index=False)


UPDATE

As per what the users have pointed out, I went ahead and tried to remedy my mistake (ignore the code repetition. I'll be solving that later):

import pandas as pd
import warnings
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

warnings.filterwarnings("ignore", category=FutureWarning)

train['Sex'] = train['Sex'].replace(['female', 'male'], [0, 1])
train['Embarked'] = train['Embarked'].replace(['C', 'Q', 'S'], [1, 2, 3])
train['Age'].fillna(train.groupby('Sex')['Age'].transform("median"), inplace=True)
train['HasCabin'] = train['Cabin'].notnull().astype(int)
train['Relatives'] = train['SibSp'] + train['Parch']
train_data = train[['Pclass', 'Sex', 'Relatives', 'Fare', 'Age', 'Embarked', 'HasCabin']]
x_train, x_validate, y_train, y_validate = train_test_split(train_data, train['Survived'], test_size=0.22, random_state=0)

test['Sex'] = test['Sex'].replace(['female', 'male'], [0, 1])
test['Embarked'] = test['Embarked'].replace(['C', 'Q', 'R'], [1, 2, 3])
test['Age'].fillna(test.groupby('Sex')['Age'].transform("median"), inplace=True)
test['HasCabin'] = test['Cabin'].notnull().astype(int)
test['Relatives'] = test['SibSp'] + test['Parch']
test_data = test[['Pclass', 'Sex', 'Relatives', 'Fare', 'Age', 'Embarked', 'HasCabin']]

logReg = LogisticRegression()
logReg.fit(x_train, y_train)

predictions = logReg.predict(test[test_data])
submission = pd.DataFrame({'PassengerId': test['PassengerId'], 'Survived': predictions})

filename = 'Titanic-Submission.csv'
submission.to_csv(filename, index=False)


As you can see, I tried to input the select test features into my algorithm

test_data = test[['Pclass', 'Sex', 'Relatives', 'Fare', 'Age', 'Embarked', 'HasCabin']]

...

predictions = logReg.predict(test[test_data])


Right now, I'm getting the following error:

Traceback (most recent call last):
predictions = logReg.predict(test[test_data])
File "C:\Users\security\Anaconda3\envs\TItanic-Kaggle.py\lib\site-packages\pandas\core\frame.py", line 2914, in __getitem__
return self._getitem_frame(key)
File "C:\Users\security\Anaconda3\envs\TItanic-Kaggle.py\lib\site-packages\pandas\core\frame.py", line 3009, in _getitem_frame
raise ValueError('Must pass DataFrame with boolean values only')
ValueError: Must pass DataFrame with boolean values only


Its telling me that I need to pass boolean values into my algorithm but I don't understand why. There wasn't such a prerequisite when I was using the exact same data format while training the model.

• As far as I can see you make changes to the xtrain DF but not to xtest. xtest must have the same logic and shape as xtrain. Otherwise the model will not know what it is supposed to predict. Do the same recoding to xtest like you did to xtrain. – Peter Jun 6 '19 at 12:32
• You only have 197 examples in your x_test set. – Cihan Dogan Jun 6 '19 at 12:44
• How am I supposed to do that? – Andros Adrianopolos Jun 6 '19 at 14:01
• You can write a function and pass dataset to it and in that function, perform all the operation that you want to perform. Then first pass the train dataset and then pass the test dataset. In this way same operations are performed on both dataset and you should not get the error which you are getting now. – vb_rises Jun 6 '19 at 14:38

Your predictions are those for x_test, which was split out from train, but your submission's PassengerIds are those from test.

It appears you want to submit predictions for test, so you need to call logReg.predict on that instead of x_test. However, as @Peter notes in a comment, that will fail since the columns in test are not the same as in train and therefore x_train and x_test. Your feature encodings, null replacement, and engineering need to be done for test as well (but take care to to use train's median when filling test's missing ages).

• So does that mean I need to repeat code? Isn't there a cleaner way of doing that? – Andros Adrianopolos Jun 6 '19 at 23:16
• For so few operations, on just two datasets, that's how I'd do it. You could do as @Vishal suggests, defining a function to apply twice. You could also combine the two frames, do some of the operations, then split them back apart (but do the imputation separately). – Ben Reiniger Jun 7 '19 at 2:52
• I took your approach. Could you check the edit if you don't mind? – Andros Adrianopolos Jun 7 '19 at 5:43
• The new error is because of test[test_data]; test_data is already a dataframe, not the list of columns. Use just logReg.predict[test_data]. – Ben Reiniger Jun 7 '19 at 14:51

OK, I see your point ... check this code snippet out

the number of predictions will be N = rows

good luck!

https://www.kaggle.com/geoffpidcock/example-submission

I feel with you that predictions/submissions are hard for beginners but at the same time, I doubt you can expect anyone here to take you step by step through the solution.