1
$\begingroup$

I have a train and test datasets (600k observations) that have different categories for the same categorical variable.

For example train has the categorical variable Letters having unique categories being {a,b,c,d,e,f,g}, while the test dataset has unique categories {a,b,c,d,e,f,h,l,m,n} for the column Letters.

I am dealing with a binary classification problem and I am mainly working with Random Forest. Clearly, {h,l,m,n} are unseen in the train dataset, so I am not sure how random forest can make accurate predictions for these data. Which turnarounds are effective in these type of problems?

I thought about clustering the data based on the correlation of other features using KMeans on the concatenated train and test and then include the Cluster category as a feature instead of Letters, however it is difficult to specify the correct number of clusters. I am new to data science so probably there are better ways to deal with this problem.

$\endgroup$
1
  • $\begingroup$ Hi kyara, Welcome to the community. Please consider upvoting/marking the answer as correct if you find anything useful on your post $\endgroup$
    – Kriti
    Dec 4, 2023 at 20:36

1 Answer 1

1
$\begingroup$

This is a very common scenario, to overcome this you can use one hot encoding for your categorical variable - sklearn.preprocessing.OneHotEncoder

One hot encoding will ensure that it gets all the categories from your training data and will encode the testing data categories according to the train data. Since your test data has new categories , it will ignore the encoding of the new categories in the test data.

from sklearn.preprocessing import OneHotEncoder
import pandas as pd

x_train=pd.DataFrame({"Categorical Variable": ["a", "b","c","d","e","f","g"]})



x_test = pd.DataFrame({"Categorical Variable": ["a","b","c","d","e","f","h","l","m","n"]})



transformer = OneHotEncoder(handle_unknown = 'ignore') ##handle_unknown plays the role of ignoring the new values in the test data
transformer.fit(x_train)
print(transformer.transform(x_train).toarray())

Here you can observe that there are only 7 one hot encoded variables; since train_data has a total of 7 categories

[[1. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0.]
 [0. 0. 0. 0. 0. 1. 0.]  
 [0. 0. 0. 0. 0. 0. 1.]]
print(transformer.categories_)

[array(['a', 'b', 'c', 'd', 'e', 'f', 'g'], dtype=object)]

print(transformer.get_feature_names())

['x0_a' 'x0_b' 'x0_c' 'x0_d' 'x0_e' 'x0_f' 'x0_g']

print(transformer.transform(x_test).toarray())

Here you can observe that the test data has also 7 categories as it comes from the training set and encoder ignores h,l,m,n

[[1. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0.]
 [0. 0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0.]]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.