3
$\begingroup$

I was wondering how can we use trained neural network model's weights or hidden layer output for simple classification problem, and then use those for feature engineering and implement some boosting algorithm on the new engineered features.

Suppose,if we have 100 rows with 5 features (100x5) matrix.

data:
      X                 Y
x1,x2,x3,x4,x5          y1  y2
0,1,2,3,4               0   1
3,2,5,6,4               1   0

Network: 2 layers, input and softmax output, compile using cross entropy .

Can we utilise trained weights or hidden layer output of above network and use it for feature engineering on original dataset and then apply some boosting algo on modified dataset and will it increase accuracy ?

Improve this question
$\endgroup$
12
  • $\begingroup$ Are you sure you mean weights? Your example network would have 12 weights in the first layer (connecting input features to the hidden layer), and 3 in the second layer (connecting hidden layer to output) - including bias terms. I think you mean activations (i.e. the outputs of the 2 neurons in the hidden layer). Could you also clarify how your network has been trained? $\endgroup$
    – Neil Slater
    Oct 23, 2017 at 12:00
  • 1
    $\begingroup$ You normally pass a lot of low level features to the NN (e.g. raw pixel values) and have the NN learn its own high level features instead of hardcoding high level features (e.g. edge detection filters). $\endgroup$
    – CodesInChaos
    Oct 23, 2017 at 12:31
  • $\begingroup$ @NeilSlater - you are right, my mistake. it would be 12 weights including bias. So, idea is to train a neural network on simple numeric data, (i'll update the question) and predict probabilities using softmax. Idea is to use neural network just for feature engineering $\endgroup$
    – CYAN CEVI
    Oct 23, 2017 at 13:38
  • $\begingroup$ @CodesInChaos - i'm not talking about image classification. simple numeric data. and i'm asking that if we can utilise those learned high level features that you mentioned to perform feature engineering on original dataset and apply some other classification algo. $\endgroup$
    – CYAN CEVI
    Oct 23, 2017 at 13:41
  • $\begingroup$ So you want to take output of NN (the predicted values of $y_1$ and $y_2$) on some examples where you have $x_1 ... x_5$ defined, and feed it into another model per example? Or maybe the output from one of the middle layers? Or do you really want to use the NN weights as a feature? $\endgroup$
    – Neil Slater
    Oct 23, 2017 at 13:41

1 Answer 1

Reset to default
2
$\begingroup$

TL;DR: Yes. You can (iiuc)

Longer Version: In fact, this is what many popular algorithms like Word2Vec and AutoEncoders do. (With respect to hidden layer outputs)

Word2Vec: Given an input word ('chicken'), the model tries to predict the neighbouring word ('wings') In the process of trying to predict the correct neighbour, the model learns a hidden layer representation of the word which helps it achieve its task.

Finally, we just remove the last layer and use the hidden layer representation of the word as its $N$ dimensional vector.

the embedded representation is what we use

So basically, we feature engineered the word vectors.

For autoencoders, It takes in $X$ as the input and tries to predict $X$ again, in the process learning a latent representation of the input signal X. The input hidden representation in Layer $L2$ can be used in other tasks. (Note: here X and X hat are the same) enter image description here

(In fact, you can use features learned by a CNN and feed them into an SVM and get good results)

Relevant to you: You can train your model on the given data and finally chop off the last prediction layer and use the output of the intermediate layers as features. I believe this should work because it works for so many tasks which I explained above.

Source:

  1. Learn Word2Vec by implementing it in Word2Vec : an article by me explaining word2vec. (Shameless self-advertising here but I feel the article is good and relevant)

  2. Andrew Ng's unsupervised feature learning website

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.