# How to use neural network's hidden layer output for feature engineering?

I was wondering how can we use trained neural network model's weights or hidden layer output for simple classification problem, and then use those for feature engineering and implement some boosting algorithm on the new engineered features.

Suppose,if we have 100 rows with 5 features (100x5) matrix.

data:
X                 Y
x1,x2,x3,x4,x5          y1  y2
0,1,2,3,4               0   1
3,2,5,6,4               1   0


Network: 2 layers, input and softmax output, compile using cross entropy .

Can we utilise trained weights or hidden layer output of above network and use it for feature engineering on original dataset and then apply some boosting algo on modified dataset and will it increase accuracy ?

• Are you sure you mean weights? Your example network would have 12 weights in the first layer (connecting input features to the hidden layer), and 3 in the second layer (connecting hidden layer to output) - including bias terms. I think you mean activations (i.e. the outputs of the 2 neurons in the hidden layer). Could you also clarify how your network has been trained? – Neil Slater Oct 23 '17 at 12:00
• You normally pass a lot of low level features to the NN (e.g. raw pixel values) and have the NN learn its own high level features instead of hardcoding high level features (e.g. edge detection filters). – CodesInChaos Oct 23 '17 at 12:31
• @NeilSlater - you are right, my mistake. it would be 12 weights including bias. So, idea is to train a neural network on simple numeric data, (i'll update the question) and predict probabilities using softmax. Idea is to use neural network just for feature engineering – CYAN CEVI Oct 23 '17 at 13:38
• @CodesInChaos - i'm not talking about image classification. simple numeric data. and i'm asking that if we can utilise those learned high level features that you mentioned to perform feature engineering on original dataset and apply some other classification algo. – CYAN CEVI Oct 23 '17 at 13:41
• So you want to take output of NN (the predicted values of $y_1$ and $y_2$) on some examples where you have $x_1 ... x_5$ defined, and feed it into another model per example? Or maybe the output from one of the middle layers? Or do you really want to use the NN weights as a feature? – Neil Slater Oct 23 '17 at 13:41

TL;DR: Yes. You can (iiuc)

Longer Version: In fact, this is what many popular algorithms like Word2Vec and AutoEncoders do. (With respect to hidden layer outputs)

Word2Vec: Given an input word ('chicken'), the model tries to predict the neighbouring word ('wings') In the process of trying to predict the correct neighbour, the model learns a hidden layer representation of the word which helps it achieve its task.

Finally, we just remove the last layer and use the hidden layer representation of the word as its $N$ dimensional vector.

So basically, we feature engineered the word vectors.

For autoencoders, It takes in $X$ as the input and tries to predict $X$ again, in the process learning a latent representation of the input signal X. The input hidden representation in Layer $L2$ can be used in other tasks. (Note: here X and X hat are the same)

(In fact, you can use features learned by a CNN and feed them into an SVM and get good results)

Relevant to you: You can train your model on the given data and finally chop off the last prediction layer and use the output of the intermediate layers as features. I believe this should work because it works for so many tasks which I explained above.

Source:

1. Learn Word2Vec by implementing it in Word2Vec : an article by me explaining word2vec. (Shameless self-advertising here but I feel the article is good and relevant)

2. Andrew Ng's unsupervised feature learning website