Aggregating small values in a frequency bar plot

Question

I have a pandas Series of sorted percentage values like this :

A -> 0.001
B -> 0.0012
C -> 0.0015
...
H -> 0.02
I -> 0.03
J -> 0.041
...
X -> 0.12
Y -> 0.31
Z -> 0.4

(I typed those by hand, I am not familiar with how to type those python outputs in DSSE, sorry! If anyone has a hint on that, it would be nice. If this pandas series would be named "content_distribution", content_distribution.X would return the float 0.12 .)

I want to obtain a Series which looks like this :

Other -> 0.07
I -> 0.03
J -> 0.041
...
X -> 0.12
Y -> 0.31
Z -> 0.4

because those small values are irrelevant to display in a plot that I want to make, and I want to keep the big values on display. Is there any pandas-ic or pythonic way to do that? I can write my own script, but anything I try out looks non-pandas-ic (even though it sort of works).

(I also don't know if "aggregating" is the correct pandas term for this, if anyone has a better one, he or she is welcome to edit my title.)

Answer 1

Here is an example where I create a new row called Other, which contains the sum of all values below a given threshold.

I then remove the rows that were below that threshold (and so included in the sum), so the final Pandas Series only has values above the threshold, plus the new Other row.

This is all performed on a randomly generated Pandas Series, as follows:

In [1]: import pandas as pd, numpy as np

In [2]: X = pd.Series(np.random.random(20))

In [3]: X
Out[3]: 
0     0.507151
1     0.366259
2     0.444196
3     0.027280    this will be in the sum
4     0.132785    this will be in the sum
5     0.993170
6     0.614712
7     0.942894
8     0.516255
9     0.107436
10    0.710416
11    0.512221
12    0.502685
13    0.753515
14    0.894735
15    0.780213
16    0.998315
17    0.971558
18    0.504525
19    0.224767
dtype: float64

In [4]: threshold = 0.2

In [5]: X.loc['Other'] = X[X < threshold].sum()

In [6]: X.drop(X[X < threshold].index, inplace=True)

In [7]: X
Out[7]: 
0        0.507151
1        0.366259
2        0.444196
5        0.993170
6        0.614712
7        0.942894
8        0.516255
10       0.710416
11       0.512221
12       0.502685
13       0.753515
14       0.894735
15       0.780213
16       0.998315
17       0.971558
18       0.504525
19       0.224767
Other    0.267501
dtype: float64

Notice that the indices still match the original values, i.e. they are not continguously numbered - indeices 3 and 4 have been dropped/removed. If you want to get continuously numbered indices back, you can do the following:

X.reset_index()

This creates a new index and puts the original one shown above in a new column called index, which can then itself be dropped if desired:

X.drop('index', axis=1)

That will remove the label Other, which was in the original index.