5
$\begingroup$

I am trying to use the MNIST dataset to train a convolutional neural network to classify digits written in colorblindness charts. As some people have suggested, I have tried playing with the brightness and contrast, as well as converting to grayscale, but all of these have inconsistent results because all charts are very different.

I am looking for general ideas about approaches I can try. Would style transfer make sense? Here is an example of an easy chart I want to be able to classify:

enter image description here

I tried various transformations and got results such as these. Although to me, a mildly colorblind person, these are easier to see, it is not nearly close to MNIST.

enter image description here

TLDR: I don’t have a dataset of color blindness charts. MNIST is readily available however. I’m trying to somehow use MNIST (probably after transforming it) to make the neural network classify the limited color blindness charts I have. MNIST and the charts are different and I need to make them similar so that the NN after training on modified MNIST can predict color blindness charts.

EDIT:

I tried the suggestions and applied various OpenCV Transformations to images. At the end, everything is resized and converted to grayscale. My accuracy went up from 11% (on non processed images) to 33% on the new images.

My problem is finding a set of transformations that are universal. My transformations work well for some images:

enter image description here

Some are not so good: enter image description here

How can I improve this? Here are the transformations I do:

image = cv2.imread(imagePath)
    image = imutils.resize(image, height=400)
    contrasted_img = CONTRASTER.apply(image, 0, 60) #applies contrast
    median = cv2.medianBlur(contrasted_img,15)
    blur_median = cv2.GaussianBlur(median,(3,3),cv2.BORDER_DEFAULT)
    clustered = CLUSTERER.apply(blur_median, 5) #k-means clustering with 5
    gray = cv2.cvtColor(clustered, cv2.COLOR_BGR2GRAY) #RESULT

SECOND EDIT:

I used written advice and calculated that the digit usually takes up between 0.1 and 0.35 ish of the image. So I increase threshold for black and white until that happens. This resulted in images like this:

enter image description here

My accuracy of the NN went up to 45%! Another amazing improvement. However, my biggest issue is with images where the digit is darker than the background:

enter image description here

This results in incorrect thresholds. I also have some noise issues, but these are less common and can be fixed with tuning my percent white and threshold:

enter image description here

$\endgroup$
8
  • $\begingroup$ Can you clarify what you mean by "I am trying to use the MNIST dataset to train a convolutional neural network to classify digits written in colorblindness charts."? Does it mean you used transfer learning? And if so: how exactly did you do it? Moreover, what size is your dataset? $\endgroup$ – Sammy Jan 14 '20 at 8:50
  • 1
    $\begingroup$ You could try to use edge detection, then you remove the outer circle using its coordinate... I dont know if it'll work but it's worth to try edge $\endgroup$ – akhetos Jan 14 '20 at 16:50
  • $\begingroup$ @Sammy I added another paragraph to summarize. Thanks! $\endgroup$ – shurup Jan 15 '20 at 15:28
  • 2
    $\begingroup$ Seems like a few preprocessing steps could get you from these charts to something closer to MNIST digits. Cluster the color values to find foreground/background color palates, remove the background colors, then blur the image slightly to get solid lines, and convert to black and white. Seems like you're trying to convert MNIST to colorblindness charts, but it might be easier to go the other way. $\endgroup$ – Nuclear Hoagie Jan 15 '20 at 15:34
  • $\begingroup$ @NuclearWang I will try that and report back. Thanks for the suggestion! $\endgroup$ – shurup Jan 16 '20 at 16:39
1
$\begingroup$

I think the $RGB$ color components of the pixels form vectors that if you see them in spherical coordinates, angles are colors, and radius is intensity.

I tried it and ended up with:

Colors to spherical angles conversion

Build a 3-class classifier for the angles calculated for each pixel (background and two colors).

If you end up with only 2 classes (the third one has very few members), you have an intensity problem, and you should reclassify for radius (or just take a threshold value).

If you have 3 classes, you have a color problem. Drop the class of the (black) background color (probably just the class of the top left pixel. Also drop the class with the largest mean distance to the center of the image, the second background.

The remaining pixels will be white, the others black.

You still have a picture with circles forming the digit, but a pooling filter will even it out.

3-way classification of the angles (R,G,B represent the classes)

3-way classification directly on the image was unsuccessful, because the distance between light gray and dark gray is similar to light gray/light purple. The angles are much more separable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.