I'm wondering if the following strategy has been already used and could work

Let's says you have a CNN which work well to classify image data, dog and cat. You only have cat and dog image as training data. Is there any way to use it to detect image of horse as anomaly?

For example, with a ruled based system we could says

if P(cat) and P(dog) ~0.5 then it's an anomaly

another way could be to take feature vector at last fully connected layer and compare vector very different from other could be considerated as anomaly

Do you have any related paper? is it a totaly dumb idea?