7
$\begingroup$

I am trying to put numeric data into fixed number of buckets using Python/R.

I have data in key:value format {1 : 12.3, 2 : 4.7, 3 : 7.4, 4 : 15.9, ......, 50 : 24.1}, which is device_id:data_usages I need to bucket based on value into nine buckets (1,5,25,50,150,250,1000,5000,10000), So later I can see which data points are in which bucket.

What function can do this in Python OR R?

$\endgroup$
2
  • $\begingroup$ Do you need to then use the data from each bucket? Or do you just want to know how many are in each bucket? $\endgroup$
    – n1k31t4
    Feb 5, 2019 at 20:55
  • $\begingroup$ Updated my question $\endgroup$
    – roy
    Feb 5, 2019 at 20:57

2 Answers 2

10
$\begingroup$

You don't really need to implement an algorithm to achieve this. There are a few tools that will do this for you.

You can get the data assigned to buckets for further processing using Pandas, or simply count how many values fall into each bucket using NumPy.

Assign to buckets

You just need to create a Pandas DataFrame with your data and then call the handy cut function, which will put each value into a bucket/bin of your definition. From the documentation:

Use cut when you need to segment and sort data values into bins.

In [1]: import pandas as pd
In [2]: import numpy as np    # to create dummy data

Create some dummy data, put it in a dataframe and define the bins:

In [3]: data = np.random.randint(low=1, high=10001, size=1000)                 
In [4]: df = pd.DataFrame(data=data, columns=["data"])
In [5]: bins = np.array([1,5,25,50,150,250,1000,5000,10000])

Pass the data, along with the bin definitions to the cut function and assign it back as a new column in the dataframe:

In [6]: df["bucket"] = pd.cut(df.data, bins)

You can then inspect the first few rows to see that the values have now been labelled with the relevant bucket:

In [7]: df.head()                                                              
Out[7]: 
   data         bucket
0  8754  (5000, 10000]
1  2970   (1000, 5000]
2  6778  (5000, 10000]
3  2550   (1000, 5000]
4  5226  (5000, 10000]

Counting how many in each bucket

Here is an example using NumPy, to get an idea of the distribution, as a histogram.

Using the data and bins as defined above, we pass them to the numpy histogram function, which will count how many data points fall into each bin:

In [8]: np.histogram(data, bins)
Out[8]: 
(array([  0,   2,   1,   8,   6,  61, 417, 505]),
 array([    1,     5,    25,    50,   150,   250,  1000,  5000, 10000]))

Where the first row tells you how many values fell into each bin, and the second row confirms the bins used.


You can get your dictionary of data into the same form as my dummy data above (into a numpy array) by doing this:

data = np.array([v for v in your_dict.values()])
$\endgroup$
2
  • $\begingroup$ Data which is in key:value format is actually device_id:data_usages. And at the end need to know which device belong to which bins $\endgroup$
    – roy
    Feb 5, 2019 at 22:11
  • 1
    $\begingroup$ @roy - then you can use the device_id as the index in the DataFrame. That way you don't lose the information. $\endgroup$
    – n1k31t4
    Feb 5, 2019 at 22:14
0
$\begingroup$

Here is a solution by using bisect Python standard library

from bisect import bisect
from random import sample

data = sample(range(10_000), 1_000)

breakpoints = [1, 5, 25, 50, 150, 250, 1_000, 5_000, 10_000]
buckets = {}

for i in data:
    buckets.setdefault(breakpoints[bisect(breakpoints, i)], []).append(i)

this will result in a dictionary with breakpoints keys and a list of data points(values) for each key.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.