How can I plot/display a dataset or an image distribution?

Question

I want to view a specific image or a dataset's distribution, and see if they are different.
Does simply writing something like :

# mydataset.shape = (50k,32,32,3)
plt.hist(mydataset.reshape(-1))

do the trick? or should I be doing something else? for example doing so on cifar10 gave me this plot :

However, it doesn't look right, there are 50K training images, and I don't know how to interpret this or even if it is the correct way of doing it in first place!

if I do :

#dataset shape is (50k, 3072)
plt.hist(mydataset.reshape(-1,32*32*3))
#and testset which is (10K,3072)
plt.hist(mytestset.reshape(-1,32*32*3))

This is what I get :

So this is very puzzling to me, and I have no idea how to go about this! Since there are 10 bins, is it giving me the distribution of each class(cifar10 as 10 classes) ? if it is so why do I get the same shape when I try to get the distribution of a single image?
I mean when I try to do :

#image is of shape (1,3072)
plt.hist(mytestset[0])

this is what I get :

and interestingly, the histogram for the whole test set looks like this :

#dataset shape is (10k,32,32,3)
plt.hist(mytestset.reshape(-1,32*32*3))

Why do I get ten bins for the single image as well?
what does each axis mean then?
What should I be looking for when it comes to image/dataset distribution?
Is it the raw values for the whole dataset only?
or Is it the raw values for each class? or even each image?

Answer 1

I want to view a specific image or a dataset's distribution, and see if they are different. Does this do the trick?

It depends what you want to understand or learn about your data.

what does each axis mean then?

In all of your plots, the x-axis ranges from 0-255, which is because in all your plots, you are creating histograms of the individual pixel values of your images. A single pixel consists of a vector (tuple) of three values: (red, green, blue). Each of those three colours can take a value 0 to 255 (usually an unsigned 8-bit integer)So by creating a histogram over your images, you are essentially counting how many time each of the possible values appear.

These plots can therefore tell you something about the average colour distribution. The peaks seem to be around 100 for each of your plots, so I guess there are a lot of mixed colours - colour where the RGB values are in the range 100 - 150. So less pixels are purely reg, green or blue e.g. (0, 0, 255) would be purely blue.

You can also compute/visualise the histogram of colours using the OpenCV library, which has great functions for doing exactly this kind of thing (and will run way faster than matplotlib's histogram method because OpenCV uses the C++ backend library). Have a look at this great walkthrough.

---

Why do I get ten bins for the single image as well?

The reason all your histograms have 10 bins, is because you are not specifying a value for the bins argument to plt.hist, so the default value is taken from the basic configurations of matplotlib, which you can see by running this:

print(plt.rcParams["hist.bins"])    # will print 10 by default

The y-axes on those two plots do make sense; the dataset with 50k images has higher counts for each pixel value compared to the 10k dataset.

---

What should I be looking for when it comes to image/dataset distribution?

Is it the raw values for the whole dataset only? or Is it the raw values for each class? or even each image?

Each plot is showing the distribution of raw values only, for whichever set of data you use. You used mtdataset and mytestset as input, so in each case you are only seeing the distribution for those images of course. There is no inclusion of the actual labels anywhere, so you are not breaking down the distributions into the target classes, for example. Just raw pixel values are considered.

---

What should I be looking for when it comes to image/dataset distribution?

This is problem specific.

You could be looking to see that there is indeed a distribution of RGB values, as a sanity check that you don't have some really skewed set of colour images that e.g. are mostly black or white. You might compare the distributions of the training and test sets to one another, to ensure that they are similar - meaning the training set is indeed representative of the test set. If this were not the case, any model you might train on a specific task could be biased towards the training set and perform badly on the test set (it will not have seen images similar to the test set if the distributions are very different!)

For training neural networks with images, it is common to normalise the distribution of pixel values to the range [-1, +1], which helps smoother learning via smoother gradient updates.