Use the output of 2 hidden neurons in the last hidden layer of a NN to visualize the result of a 4-class classification task

Question

I am working on a 4-class classification task using an Artificial Neural Network. My aim is to visualize how well those 4 classes can be separated, moreover how consistent the data coming from each class is and how far they are "away" from one another.

My approach is to use 2 neurons in the last hidden layer and plot the output of the neurons in a scatterplot. However my current output usually looks like this where the different colors stand for the actual classes the data is coming from:

Obviously this is not very useful. First of all I was hoping that the points would not all be all along the same line and second of all I am not sure if this can even be interpreted in the way that I want.

I am using the neuralnet package in R to train the network using linear output. I am guessing that if this approach can work the choice of activation function would be crucial.

I am new to Neural Networks and I am not sure whether or not something like this can be done or not and I am hoping to get some comments.

Answer 1

I'm somewhat new to the topic as well, but I think what you are looking for is an autoencoder. I have only used it in the h2o deeplearning package, but it seems to work well.

The idea behind an auto-encoder is to begin with your inputs, encode down to less nodes (two is nice for visual representation), then decode back to your original outputs (ass accurately as possible).

If the auto-encoder can decode back to an accurate representation of your inputs, then there was sufficient important information withheld in those two nodes. You can then plot the features that those two nodes held in a two-dimensional plot that would be analogous to a bounded PCA plot.

Also - the auto-encoder is not constrained to linearity like the PCA is. Below is a quick snippet of some code, hope it helps.

library(h2o)
localH2O = h2o.init(ip = "localhost", port = 54321, startH2O = TRUE, min_mem_size = "3g", max_mem_size = "4g", nthreads = -1)
dat_h2o <- as.h2o(train.df)

unsupervised <- 
  h2o.deeplearning(x = 24:356,  # column numbers to use
                   training_frame = dat_h2o, # data in H2O format
                   autoencoder = TRUE, ## unsupervised autoencoding
                   activation = "Tanh", # or 'Tanh' 'Rectifier' 'WithDropout' node activation function, Tanh seems to work best for autoencoding
                   hidden = c(5,2,5), # three layers of nodes, with 5/2/5 nodes, respectively
                   epochs = 1) # max. no. of epochs 

## layer 1 corresponds to hidden[1], so it will reduce to 5 variables
## below is roughly similar to using predict(pca.object, newdata)
training_data <- h2o.deepfeatures(unsupervised, dat_h2o, layer = 1)
testing_data <- h2o.deepfeatures(unsupervised, test_h2o, layer = 1)

## explore the second layer with 2 nodes, similar to exploring PC1 vs PC2
train_supervised_features2 = h2o.deepfeatures(unsupervised, dat_h2o, layer=2)

plotdata2 = as.data.frame(train_supervised_features2)
plotdata2$label = as.character(as.vector(dat_h2o[,364]))

## L2 corresponds to layer 2, so use L2
qplot(DF.L2.C1, DF.L2.C2, data = plotdata2, color = label, main = 'Neural network: 5-2-5')

Answer 2

First of all, I would think that plot is useful. The classes do seen fairly separable in that space.

A first pass option for visualization should be PCA, though as pointed out in other answers, it will consider only linear combinations of features.

If you want to visualize the separability of the classes in the context of how well your classifier is doing, the plot you made is a good option. It shows that while there is some confusion (for green in particular), you could fairly easily group each class on that plot.

If instead you want to visualize the natural separability of the classes, PCA or a low-dimensional autoencoder is the way to go.

The difference is that if what you care about is the former, it's a good thing that your representation is optimized to separate the classes. If you care about the latter, the unsupervised nature of autoencoders is more appropriate.