2
$\begingroup$

Using tensorflow. If I have trained my image classifier many images of a close up of melanomas and other skin cancers.

Then ask it to classify using the trained model a photo that has another element for example melanoma on a hand (with other elements that look like melanoma in the background example a hypothetical tablecloth that has melanoma patterns).

Is there a way to visualize what the image classifier has identified as melanoma ? Outputting a circled/highlighted the found melanomas in the photo?

Improve this question
$\endgroup$
  • $\begingroup$ I am not so sure but you need to look at CAM's probably $\endgroup$ – Aditya Aug 13 '18 at 12:26
1
$\begingroup$

This question was always in my mind when I started to learn deep learning. The answer is what I understood:

In the training phase: actually, when we train our model it learns feature from images by updating filter weights i.e parameters through back-propagation. And this trained model learns to generalize feature of the input images.

In testing phase: with the help of updated weights, now our model can understand (what image is this) the input image.

If you want to get more details clarification, you may visit these links:

Link1 [Specially- step 4]
Link2 Link3

Improve this answer
$\endgroup$
  • $\begingroup$ How does this answer the question: Is there a way to visualize what the image classifier has identified as melanoma? $\endgroup$ – Stephen Rauch♦ Apr 13 '19 at 1:02
1
$\begingroup$

What you are looking for is a model with an "Attention mechanism". Unfortunately, this means that you have to change your model and to retrain it. It is more complicated that a simple image classifier but I can assure you that it is worth it to learn how it works. Or you can simply find an existing repository that implement it.

Alternatively, you can train your model on smaller images that contain exactly the melanoma. The preprocessing will be much heavier. Then you will scan your bigger images with a bounding box this is known as sliding window. You will have probabilities for each small images.

Sliding Window (Formally):

The propose is: Given a classifying model $F(J)$ where $J$ are images of size $w \times h$ you will apply this model to a image $I[m,n]$ by slicing it into many blocks $B[m,n]$ where $$ B[m,n] = I\Biggl[\Biggl(m-\frac{w}{2}:m+\frac{w}{2}\Biggr),\Biggl(n-\frac{h}{2}:m+\frac{h}{2}\Biggl)\Biggr]$$

And you will apply the model as $F(B[m,n]), \forall m \in [w,W-w] and n \in [h,H-h]$ where $W$ and $H$ are the width and height of $I$.

Important to notice that for OpenCV images the height is actually the first coordinate of the variable (numpy array).

If you have a mean size of melanoma images you can run through $m$ and $n$ with steps of half of the mean size.

So you might search for which parts of the image where classified as melanoma by looking into $$H[m,n] = F(B[m,n])$$ and using it as a binary mask to identify melanoma classified image parts.

Improve this answer
$\endgroup$
  • $\begingroup$ What do you mean by "Then you will scan your bigger images with a bounding box"? $\endgroup$ – Azeworai Aug 15 '18 at 12:03
  • $\begingroup$ You scan small parts of the new image (the parts inside a moving bounding box) the subimage with the highest probability is the one where the bounding box is on the melanoma. $\endgroup$ – bukwyrm Sep 12 '18 at 15:33
0
$\begingroup$

Actually you don't need to retrain the model, you could use deconvolution networks (see this video of Matthew Zeiler explaining this paper: Visualizing and Understanding Deep Neural Networks) or get inspiration with this paper: Methods for Interpreting and Understanding Deep Neural Networks.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.