1
$\begingroup$

In this article, the author creates a graph (at the end of the post) from the embeddings of different words found by transformer model. I would like to do a similar thing for a convolutional neural network in order to be able to evaluate clusters. The final objective is to be able to identify similar images in the train set to a given image.

I thought about extracting the hidden representation created by one of the hidden layers and reduce the dimensions to 2 using something like PCA.

I have some doubts:

  • Is this strategy sound?
  • Which layer should I use? Should I use the last one, as when creating a Global Class Activation map?

scatter plot

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, the approach you propose is sound and applied widely.

Instead of PCA, I suggest using U-MAP, which will probably yield better results (also better than t-SNE).

The representation you may use as input to U-MAP is the output last layer before the projection to the label space dimensionality (e.g. with a 5-class classifier, you would take the vector representation before projection to the 5-dimensional space).

$\endgroup$
3
  • $\begingroup$ Thank you, that is very helpful. I should take the output layer before without multiplying by the classifier weights right? $\endgroup$
    – Zan
    Feb 8, 2023 at 23:20
  • $\begingroup$ Yes, that's it. $\endgroup$
    – noe
    Feb 9, 2023 at 7:21
  • 1
    $\begingroup$ Accepted, thank you for your help! $\endgroup$
    – Zan
    Feb 9, 2023 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.