1
$\begingroup$

I have pandas dataframe, and I want to bin the continuous values.

a['abc'].describe() # a name of pandas dataframe, abc--column name
count    250000.000000
mean         43.412040
std          26.075295
min           0.000000
25%          25.000000
50%          38.000000
75%          53.000000
max         218.000000
Name: abc, dtype: float64

On using pandas qcut for 4 groups, how is a negative value assigned in one of bins?

a["abc_bin"] = pd.qcut(a["abc"],4,labels=None,)
print(a["abc_bin"].value_counts())

(25.0, 38.0]      73448
(-0.001, 25.0]    62818
(53.0, 218.0]     61605
(38.0, 53.0]      52129
Name: abc_bin, dtype: int64

How is the bin width is decided? In particular, how is there a negative value as a bin edge?

$\endgroup$

1 Answer 1

1
$\begingroup$

Why does one bin include negative values?

This is because the resulting intervals are open on the left, so pandas extends the left edge to include the min. Based on your describe output, the min is 0, so the left edge becomes slightly negative:

  • (0, 25.0] does not include 0
  • (-0.001, 25.0] includes 0

This is not documented in qcut, but similar behavior is explained in cut:

bins: ... The range of x is extended by 0.1% on each side to include the minimum and maximum values of x.


How are the bins determined?

qcut adjusts the edges such that each bin contains the same number of elements, whereas cut just divides strictly at the edges.

  • So with cut, we can avoid the negative edge by specifying a list of bins because the data gets split exactly at those edges:

    a = pd.DataFrame({'abc': np.random.random(size=10000)})
    pd.cut(a['abc'], [0, 0.25, 0.5, 0.75, 1]).value_counts()
    
    # (0.25, 0.5]    2637
    # (0.0, 0.25]    2478
    # (0.75, 1.0]    2454
    # (0.5, 0.75]    2431
    # Name: abc, dtype: int64
    
  • But with qcut, this workaround has no effect since qcut always adjusts the edges to force the bins into equal counts:

    pd.qcut(a['abc'], [0, 0.25, 0.5, 0.75, 1]).value_counts()
    
    # (-0.000818, 0.253]    2500
    # (0.253, 0.489]        2500
    # (0.489, 0.745]        2500
    # (0.745, 1.0]          2500
    # Name: abc, dtype: int64
    
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.