# How to assign a new level to many levels of a categorical variable

I have a model which has many categorical variables. For each categorical variable there are many levels, like 50~. But not all of them have significant counts. I got these counts using the function value_counts() in Python:

A                 50
B                 38
C                 26
D                 18
E                 10
...
T                 1
X                 1
Z                 1


How can I change the levels with count (say) less than 5 to a new level "others"?

for x in data.class:
if x.value_counts() <30:
x = "others"

• If would be useful if you could provide what the original data look like. It is unclear whether you want to act on the original data frame or on the one that you obtain after the value_counts(). In the latter case you could just rename the entries that satisfy your condition: what in particular have you found difficult to achieve? – gented Aug 18 '16 at 19:02

Using the notation you gave (i.e. data is the dataframe and data.class is the categorical variable you want to process), this can be done this way:

def cut_levels(x, threshold, new_value):
value_counts = x.value_counts()
labels = value_counts.index[value_counts < threshold]
x[np.in1d(x, labels)] = new_value

cut_levels(data.class, 30, 'others')


Note that this modifies the original data, thus if you want to keep it, do a copy of it before or change the code to this:

def cut_levels(x, threshold, new_value):
x = x.copy()
value_counts = x.value_counts()
labels = value_counts.index[value_counts < threshold]
x[np.in1d(x, labels)] = new_value
return x

new_class = cut_levels(data.class, 30, 'others')

• Instead of using np.in1d(x, labels) it is better to use x.isin(labels). It is way faster. – Tonca Apr 8 '19 at 16:28

Say your data is in a DataFrame df, and the column of interest is named column.

treshold = 5 #or whatever value makes sense to you

def set_levels(df):

for i df.column.unique():
if len(df.loc[df.column == i]) < treshold:
df.loc[df.column == i] = 'other'

return df

new_df = set_levels(df)

• @Murtuza_07 Let me know if the solution works. In that case, if you could accept the answer I'd be very thankful. – Carlo Mazzaferro Aug 17 '16 at 21:10
• What is the purpose of the counter count? – gented Aug 18 '16 at 19:04