In Machine Learning Kernels on Kaggle I often see EDAs with structured data. So, I was wondering, if there are any recommended/standard procedures for EDA with image datasets. What kind of statistical analyses do you conduct, what kind of plots do you draw and what aims do you have in mind?
Have a look at The HASYv2 dataset. I tried to do as much of the exploratory work as possible to make sure that others can directly try more interesting thing with the dataset.
Image format specific stuff
- (Minimum / Median / Mean / Maximum) (width / height / area)
- Image formats
- Exiff meta data
For this kind of stuff, you might want to have a look at
Image/ML specific stuff
Things you can do with images:
You can compute the correlation of pixels, e.g. Figure 3:
- Plot the distribution of classes.
- Behaviour of standard classification algorithms (CNNs, VGG-16)
- Confusion Matrix Ordering (page 48 - 52, especially Figure 5.12 and 5.13): Find similar classes
Since we are talking about visual data, I would suggest to perform a clustering of images features extracted from a pre-trained neural network on similar images for e.g. if its camera images model trained on imagenet, if its CG (Computer Generated Images e.g. cartoons) a model trained on similar dataset, and perform a T-SNE visualization, and visually examine the clusters. That can be a way to perform EDA on a image dataset.
Example Image of T-SNE on Images Dataset: Link
Idea from Lev Manovich, shown on video shows how most effectively plotting picture data to chart is made by showing the images as the dots in the chart.
When something interesting behavior is on one point, you can observe the image right away.
There was an example about plotting time series data on x-y space and from the form of daily curve there was a anomality in one day. Directly zooming on the pictures on that revealed a fire taken place in the landscape. From average value black and white plots you would not get that kind of immediate observations.
There are so many ways to approach EDA for CV-based models, since there are so many dimensions to the problems CV models can solve. I like to break the first step down into two categories:
Annotation Metrics: What is the distribution of classes in your dataset? Which classes are overrepresented and which are underrepresented? Do all instances of a class share the same location and orientation within the dataset, or are they varied? This kind of EDA most often looks like a histogram or pie chart of classes in a dataset (below is a pie chart showing the classes present in the Coco val 2017 dataset); this makes it really easy to see which classes are out of alignment with their occurrences in the field. Once you know the answer to this question you can collect more data or augment what you have.
Image Metrics: What kind of images are you training your model on? Are the conditions of those images (brightness, dimensions, resolution) the same as what you'll get in the field? EDA for more classical computer vision metrics can look like a scatter plot, bar chart, or really any visualization technique you would use for generic EDA, since image metrics boil down to numbers just like any other statistic. Image metrics are fairly standard, as computer vision has been around for much longer than fancy ML techniques. Below is an example of a scatter plot of pixel values for the blue channel in the Coco val 2017 dataset - it’s clear to see where most images are clustered, and where the outliers are.
EDA for computer vision is just like EDA for any other domain - the hard part is understanding the metrics that are unique to image processing and annotations before diving into EDA. Once you have a good understanding of those two branches of analysis, it's easier to apply classical EDA techniques to large datasets of images and annotations. For a slightly more generalizable look at EDA for CV applications, check out this blog I wrote for work. EDA is so powerful because it can help generate actionable insights that will make your final solution more robust once it gets deployed.