1
$\begingroup$

I am trying to build a decision tree model. After one-hot encoding, it seems somehow the data still has a problem. When I run the following code, I receive this error:

ValueError: Number of labels=172 does not match number of samples=540


#Code:  
import pandas as pd
import numpy as np

df = pd.read_csv("https://library.startlearninglabs.uw.edu/DATASCI420/Datasets/Bank%20Data.csv", sep=",")
df.columns=['age', 'sex', 'region', 'income', 'married', 'children', 'car',
       'save_act', 'current_act', 'mortgage', 'pep']
df.info()


(nrows, ncols) = df.shape
colnames = list(df.columns.values)
string_encoding = {}
df_encoded = df.copy()
for i in range(ncols):
    levels = list(set(df.iloc[:, i]))
    num_levels = len(levels)
    string_encoding_i = dict(zip(levels, range(num_levels)))
    string_encoding[colnames[i]] = string_encoding_i
    for j in range(nrows):
        df_encoded.iloc[j, i] = string_encoding_i[df.iloc[j, i]]

print(string_encoding)
print(df_encoded.head())

# One Hot Encoding Categorial Variables
from sklearn import preprocessing
enc = preprocessing.OneHotEncoder()

non_categorial_features = ['age',
                          'income',
                          'children',
                          'pep']

for categorical_feature in list(df.columns):
    if categorical_feature not in non_categorial_features:
        df[categorical_feature] = df[categorical_feature].astype('category')

df_with_dummies = pd.get_dummies(df, sparse=True)

df = pd.concat([df, df_with_dummies], axis=1)

df.head(5)

df = df.drop(['sex', 'region', 'married', 'car',
       'save_act', 'current_act', 'mortgage', 'pep_NO', 'pep_YES'], axis=1)

df.head()
df.info()

from sklearn import tree
import numpy as np
from sklearn.model_selection import train_test_split 

# prepare for decision tree
np.random.seed(101)

title_names =['age', 'income', 'children', 'counts', 'age', 'income',
       'children', 'sex_FEMALE', 'sex_MALE', 'region_INNER_CITY',
       'region_RURAL', 'region_SUBURBAN', 'region_TOWN', 'married_NO',
       'married_YES', 'car_NO', 'car_YES', 'save_act_NO', 'save_act_YES',
       'current_act_NO', 'current_act_YES', 'mortgage_NO', 'mortgage_YES', 'pep']

df = df[title_names]


X = df.iloc[:,0:23]
Y = df.iloc[:, 23]
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.1, random_state = 99)

# decision tree
from sklearn.tree import DecisionTreeClassifier 

# Use entropy = no limit on samples for split
model_ent = DecisionTreeClassifier(criterion='entropy').fit(X_train, y_train) 
y_ent_pred = model_ent.predict(X_test)

# Use information gain (default) limit min_samples to 4
model_gini = DecisionTreeClassifier(min_samples_leaf=5).fit(X_train, y_train)
y_gini_pred = model_gini.predict(X_test)
```
$\endgroup$
  • $\begingroup$ here is the error info: $\endgroup$ – user633599 Feb 10 '19 at 19:00
  • $\begingroup$ ValueError: Number of labels=150 does not match number of samples=480 $\endgroup$ – user633599 Feb 10 '19 at 19:01
1
$\begingroup$

It looks like Y is a SpareSeries as well as y_train and y_test. So when that is passed to the decision tree fit method, it only interprets those entries with label 1 as existing. According to the pandas documentation:

We have implemented “sparse” versions of Series and DataFrame. These are not sparse in the typical “mostly 0”. Rather, you can view these objects as being “compressed” where any data matching a specific value (NaN / missing value, though any value can be chosen) is omitted.

I'm not sure why it is a sparse data structure, but you can use the to_dense method to densify it:

Y = df.iloc[:, 23].to_dense()

Edit: Danny below mentions you could just remove Sparse=True from get_dummies.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I agree with you but since it's quite a small dataset, I'd recommend to get rid of the sparse = True. $\endgroup$ – Danny Feb 11 '19 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.