I want to view a specific image or a dataset's distribution, and see
if they are different. Does this do the trick?
It depends what you want to understand or learn about your data.
what does each axis mean then?
In all of your plots, the x-axis ranges from 0-255
, which is because in all your plots, you are creating histograms of the individual pixel values of your images. A single pixel consists of a vector (tuple) of three values: (red, green, blue)
. Each of those three colours can take a value 0
to 255
(usually an unsigned 8-bit integer)So by creating a histogram over your images, you are essentially counting how many time each of the possible values appear.
These plots can therefore tell you something about the average colour distribution. The peaks seem to be around 100 for each of your plots, so I guess there are a lot of mixed colours - colour where the RGB values are in the range 100 - 150. So less pixels are purely reg, green or blue e.g. (0, 0, 255) would be purely blue.
You can also compute/visualise the histogram of colours using the OpenCV library, which has great functions for doing exactly this kind of thing (and will run way faster than matplotlib
's histogram method because OpenCV uses the C++ backend library). Have a look at this great walkthrough.
Why do I get ten bins for the single image as well?
The reason all your histograms have 10 bins, is because you are not specifying a value for the bins
argument to plt.hist
, so the default value is taken from the basic configurations of matplotlib
, which you can see by running this:
print(plt.rcParams["hist.bins"]) # will print 10 by default
The y-axes on those two plots do make sense; the dataset with 50k images has higher counts for each pixel value compared to the 10k dataset.
What should I be looking for when it comes to image/dataset distribution?
Is it the raw values for the whole dataset only? or Is it the raw values for each class? or even each image?
Each plot is showing the distribution of raw values only, for whichever set of data you use. You used mtdataset
and mytestset
as input, so in each case you are only seeing the distribution for those images of course. There is no inclusion of the actual labels anywhere, so you are not breaking down the distributions into the target classes, for example. Just raw pixel values are considered.
What should I be looking for when it comes to image/dataset
distribution?
This is problem specific.
You could be looking to see that there is indeed a distribution of RGB values, as a sanity check that you don't have some really skewed set of colour images that e.g. are mostly black or white. You might compare the distributions of the training and test sets to one another, to ensure that they are similar - meaning the training set is indeed representative of the test set. If this were not the case, any model you might train on a specific task could be biased towards the training set and perform badly on the test set (it will not have seen images similar to the test set if the distributions are very different!)
For training neural networks with images, it is common to normalise the distribution of pixel values to the range [-1, +1]
, which helps smoother learning via smoother gradient updates.