I am trying to solve the San Francisco Crime Problem on Kaggle. To begin with, here is my code:

import numpy as np
import pandas as pd
from sklearn import preprocessing
from pandas import read_csv
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import log_loss

data = pd.read_csv('/home/limafoxtrottango/Downloads/sf_crime_train.csv')
data_test = pd.read_csv('/home/limafoxtrottango/Downloads/Mozilla Downloads/Test.csv')

#pre-processing training data
le = LabelEncoder()
data['Category'] = le.fit_transform(data['Category'])

categorical_variables = pd.get_dummies(data[['DayOfWeek','PdDistrict','Resolution']])  

data = data.join(categorical_variables)

scaler1 = StandardScaler()
scaler2 = StandardScaler()
scaler3 = StandardScaler()
scaler4 = StandardScaler()

data['X'] = scaler1.fit_transform(data[['X']])
data['Y'] = scaler2.fit_transform(data[['Y']])

data['Date'], data['Time'] = data['Dates'].str.split(' ', 1).str
data['Year'], data['Month'], data['Day'] = data['Date'].str.split('-', 2).str
data['Hours'], data['Minutes'], data['Seconds'] = data['Time'].str.split(':', 2).str
data["Hours"] = data["Hours"].map(str) + data["Minutes"]

del data['Minutes']
del data['Seconds']
del data['Time']
del data['Day']
del data['Year']
del data['Id']
del data['Address']
del data['Descript']
del data['PdDistrict']
del data['Resolution']
del data['DayOfWeek']
del data['Date']
del data['Dates']

data['Month'] = scaler1.fit_transform(data[['Month']])
data['Hours'] = scaler2.fit_transform(data[['Hours']])

labels = data.columns[1:]
train = data.loc[:, labels].values

#doing pca
pca = PCA(n_components=3)
principalComponents = pca.fit_transform(train)
principalDf = pd.DataFrame(data = principalComponents, columns = ['pc 1', 'pc 2', 'pc 3'])

final_data_train = pd.concat([data[['Category']], principalDf], axis = 1)

target = np.array(final_data_train['Category'])
features = final_data_train.drop('Category', axis = 1)

# Saving feature names for later use
feature_list = list(final_data_train.columns)

# Convert to numpy array
features = np.array(features)

# Instantiate model with 10 decision trees
rf = RandomForestClassifier(n_estimators = 10, random_state = 0)

# Train the model on training data
rf.fit(features, target);

#pre-processing testing data
categorical_variables = pd.get_dummies(data_test[['DayOfWeek','PdDistrict','Resolution']])  

data_test = data_test.join(categorical_variables)

scaler5 = StandardScaler()
scaler6 = StandardScaler()
scaler7 = StandardScaler()
scaler8 = StandardScaler()

data_test['X'] = scaler5.fit_transform(data_test[['X']])
data_test['Y'] = scaler6.fit_transform(data_test[['Y']])

data_test['Date'], data_test['Time'] = data_test['Dates'].str.split(' ', 1).str
data_test['Year'], data_test['Month'], data_test['Day'] = data_test['Date'].str.split('-', 2).str
data_test['Hours'], data_test['Minutes'], data_test['Seconds'] = data_test['Time'].str.split(':', 2).str
data_test["Hours"] = data_test["Hours"].map(str) + data_test["Minutes"]

row_id_test = data_test['Id']

del data_test['Minutes']
del data_test['Seconds']
del data_test['Time']
del data_test['Day']
del data_test['Year']
del data_test['Address']
del data_test['PdDistrict']
del data_test['Resolution']
del data_test['DayOfWeek']
del data_test['Date']
del data_test['Dates']

data_test['Month'] = scaler7.fit_transform(data_test[['Month']])
data_test['Hours'] = scaler8.fit_transform(data_test[['Hours']])

data_test_trans = pca.transform(data_test)

predictions = rf.predict_proba(data_test_trans)

final_predictions_file = pd.concat([row_id_test, pd.DataFrame(predictions)], axis = 1)

np.savetxt("predictions.csv", final_predictions_file, delimiter=",")

I have done PCA on my training data, and using that object to transform the testing data using: pca.transform(). But I am getting the following error:

ValueError: operands could not be broadcast together with shapes (60002,39) (38,) during pca.transform

Can someone please point out what I am doing wrong? When I use the training data itself as testing data, and do not remove the target variable from the dataframe, the program runs just fine. When I use the actual testing data, that does not have any target column, this error is thrown.

I have just started with ML, so please excuse if this question appears a bit too naive. Thanks!

  • $\begingroup$ which line has this error? but clearly the answer is that are doing an operation that the demension of an array is 38, it should be 39. are familiar with broadcasting? $\endgroup$ – Media Apr 23 '18 at 11:56
  • $\begingroup$ This one: data_test_trans = pca.transform(data_test) $\endgroup$ – Prashant Pandey Apr 23 '18 at 11:56
  • $\begingroup$ what are the dimensions? are 39 and 38 related? are you familiar with broadcasting? $\endgroup$ – Media Apr 23 '18 at 11:58
  • $\begingroup$ There are 39 categories of crimes. From 0 to 38. $\endgroup$ – Prashant Pandey Apr 23 '18 at 12:01
  • $\begingroup$ Do you need to have a look at the test and train CSV files (after processing)? $\endgroup$ – Prashant Pandey Apr 23 '18 at 12:02

This type of error is generally observed when the training data used for preparing a model/system has a different dimension than the data used for prediction.

In this case training data dimension is (x,38) whereas testing data dimension is (y,39).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.