Tips on how to create box whisker plot for huge data set

Question

I have a huge dataset (about 2 million lines) that I want to visualize to have an idea of how spread the data is. The problem now is when I create a box whisker plot the resulting graph is not legible due to the huge amount of data.

Is there any trick to be able to successfully create a box whisker plot for huge data set in the way that is readable?

Answer 1

Hope you are aware of data sampling methods. They exist to solve problems such as these. They are of several types like;

Probability Sampling Methods

1. Simple random sampling
2. Systematic sampling
3. Stratified sampling
4. Clustered sampling

Non-Probability Sampling Methods

1. Convenience sampling
2. Quota sampling
3. Judgement (or Purposive) Sampling
4. Snowball sampling

When sampling the data, be careful of the bias.

Bias in sampling

There are five important potential sources of bias that should be considered when selecting a sample, irrespective of the method used. Sampling bias may be introduced when:

1. Any pre-agreed sampling rules are deviated from
2. People in hard-to-reach groups are omitted
3. Selected individuals are replaced with others, for example if they are difficult to contact
4. There are low response rates
5. An out-of-date list is used as the sample frame (for example, if it excludes people who have recently moved to an area)

Basis of the above detail, I think there is no need to use the entire dataset for visualization purpose. Consider an example, each country conducts a census survey to estimate its population of people. Such a dataset is huge both in size and in complexity. Do you think those statisticians use the complete dataset for visualization? I strongly doubt it. They use sampling methods.

Edit

Python Code

# load required libraries
import pandas as pd
# create some dummy data
df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                   'num_wings': [2, 0, 0, 0],
                   'num_specimen_seen': [10, 2, 1, 8]},
                  index=['falcon', 'dog', 'spider', 'fish'])
print("## Original data ##\n",df)

## Original data ##
         num_legs  num_wings  num_specimen_seen
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1
fish           0          0                  8

Sampling methods:

1. Simple random sampling: Extract 3 random rows

    print("\n Simple random sample")
    print(df.sample(3, random_state=10))
    Simple random sample
            num_legs  num_wings  num_specimen_seen
    spider         8          0                  1
    falcon         2          2                 10
    fish           0          0                  8

1.1. Simple random sampling: Extract 3 random elements from the Series

df['num_legs']
print("\n Simple random sample of particular column\n",df['num_legs'].sample(n=3, random_state=10))
 
Simple random sample of particular column
spider    8
falcon    2
fish      0
Name: num_legs, dtype: int64

1.2. Random sampling with replacement

print("\nA random 50% sample of the DataFrame with replacement:")
print(df.sample(frac=0.5, replace=True, random_state=10))
A random 50% sample of the DataFrame with replacement:
     num_legs  num_wings  num_specimen_seen
dog         4          0                  2
dog         4          0                  2

1.3. Random upsampling with replacement

print("\nAn upsample sample of the DataFrame with replacement")
print(df.sample(frac=2, replace=True, random_state=10))
An upsample sample of the DataFrame with replacement
        num_legs  num_wings  num_specimen_seen
dog            4          0                  2
dog            4          0                  2
falcon         2          2                 10
fish           0          0                  8
falcon         2          2                 10
dog            4          0                  2
fish           0          0                  8
falcon         2          2                 10

1.4. Random sampling with weights

print("\nUsing a DataFrame column as weights. Rows with larger value in the num_specimen_seen column are more likely to be sampled.")
print(df.sample(n=2, weights='num_specimen_seen', random_state=10))
# Using a DataFrame column as weights. Rows with larger value in the num_specimen_seen column are more likely to be sampled.
        num_legs  num_wings  num_specimen_seen
fish           0          0                  8
falcon         2          2                 10

2. Stratified Random Sampling

    print("\nStratified Random Sampling")
    print(df.groupby('num_legs', group_keys=False).apply(lambda x:x.sample(min(len(x), 2))))

Stratified Random Sampling

        num_legs  num_wings  num_specimen_seen
fish           0          0                  8
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1

This brief example should get you started!