Cutting numbers into fixed buckets

Question

I am trying to put numeric data into fixed number of buckets using Python/R.

I have data in key:value format {1 : 12.3, 2 : 4.7, 3 : 7.4, 4 : 15.9, ......, 50 : 24.1}, which is device_id:data_usages I need to bucket based on value into nine buckets (1,5,25,50,150,250,1000,5000,10000), So later I can see which data points are in which bucket.

What function can do this in Python OR R?

Answer 1

You don't really need to implement an algorithm to achieve this. There are a few tools that will do this for you.

You can get the data assigned to buckets for further processing using Pandas, or simply count how many values fall into each bucket using NumPy.

Assign to buckets

You just need to create a Pandas DataFrame with your data and then call the handy cut function, which will put each value into a bucket/bin of your definition. From the documentation:

Use cut when you need to segment and sort data values into bins.

In [1]: import pandas as pd
In [2]: import numpy as np    # to create dummy data

Create some dummy data, put it in a dataframe and define the bins:

In [3]: data = np.random.randint(low=1, high=10001, size=1000)                 
In [4]: df = pd.DataFrame(data=data, columns=["data"])
In [5]: bins = np.array([1,5,25,50,150,250,1000,5000,10000])

Pass the data, along with the bin definitions to the cut function and assign it back as a new column in the dataframe:

In [6]: df["bucket"] = pd.cut(df.data, bins)

You can then inspect the first few rows to see that the values have now been labelled with the relevant bucket:

In [7]: df.head()                                                              
Out[7]: 
   data         bucket
0  8754  (5000, 10000]
1  2970   (1000, 5000]
2  6778  (5000, 10000]
3  2550   (1000, 5000]
4  5226  (5000, 10000]

Counting how many in each bucket

Here is an example using NumPy, to get an idea of the distribution, as a histogram.

Using the data and bins as defined above, we pass them to the numpy histogram function, which will count how many data points fall into each bin:

In [8]: np.histogram(data, bins)
Out[8]: 
(array([  0,   2,   1,   8,   6,  61, 417, 505]),
 array([    1,     5,    25,    50,   150,   250,  1000,  5000, 10000]))

Where the first row tells you how many values fell into each bin, and the second row confirms the bins used.

---

You can get your dictionary of data into the same form as my dummy data above (into a numpy array) by doing this:

data = np.array([v for v in your_dict.values()])

Answer 2

Here is a solution by using bisect Python standard library

from bisect import bisect
from random import sample

data = sample(range(10_000), 1_000)

breakpoints = [1, 5, 25, 50, 150, 250, 1_000, 5_000, 10_000]
buckets = {}

for i in data:
    buckets.setdefault(breakpoints[bisect(breakpoints, i)], []).append(i)

this will result in a dictionary with breakpoints keys and a list of data points(values) for each key.