0
$\begingroup$

I have a model which has many categorical variables. For each categorical variable there are many levels, like 50~. But not all of them have significant counts. I got these counts using the function value_counts() in Python:

A                 50 
B                 38
C                 26
D                 18
E                 10
...
T                 1
X                 1
Z                 1

How can I change the levels with count (say) less than 5 to a new level "others"?

for x in data.class:
    if x.value_counts() <30:
        x = "others"
$\endgroup$
1
  • $\begingroup$ If would be useful if you could provide what the original data look like. It is unclear whether you want to act on the original data frame or on the one that you obtain after the value_counts(). In the latter case you could just rename the entries that satisfy your condition: what in particular have you found difficult to achieve? $\endgroup$ – gented Aug 18 '16 at 19:02
0
$\begingroup$

Using the notation you gave (i.e. data is the dataframe and data.class is the categorical variable you want to process), this can be done this way:

def cut_levels(x, threshold, new_value):
    value_counts = x.value_counts()
    labels = value_counts.index[value_counts < threshold]
    x[np.in1d(x, labels)] = new_value

cut_levels(data.class, 30, 'others')

Note that this modifies the original data, thus if you want to keep it, do a copy of it before or change the code to this:

def cut_levels(x, threshold, new_value):
    x = x.copy()
    value_counts = x.value_counts()
    labels = value_counts.index[value_counts < threshold]
    x[np.in1d(x, labels)] = new_value
    return x

new_class = cut_levels(data.class, 30, 'others')
$\endgroup$
1
  • $\begingroup$ Instead of using np.in1d(x, labels) it is better to use x.isin(labels). It is way faster. $\endgroup$ – Tonca Apr 8 '19 at 16:28
0
$\begingroup$

Say your data is in a DataFrame df, and the column of interest is named column.

treshold = 5 #or whatever value makes sense to you

def set_levels(df):

    for i df.column.unique():
       if len(df.loc[df.column == i]) < treshold:
          df.loc[df.column == i] = 'other'

    return df

new_df = set_levels(df)
$\endgroup$
2
  • $\begingroup$ @Murtuza_07 Let me know if the solution works. In that case, if you could accept the answer I'd be very thankful. $\endgroup$ – Carlo Mazzaferro Aug 17 '16 at 21:10
  • $\begingroup$ What is the purpose of the counter count? $\endgroup$ – gented Aug 18 '16 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.