7
$\begingroup$

An interviewer told me that we cannot concatenate an embedding from a neural network (such as a pre-trained image representation) and hand designed features (such as image metadata) for use in a linear model such as logistic regression. He says they must only be used with neural networks.

I understand the resulting data may not be linearly separable but that to me seems like a case-by-case empirical question.

Is he right? Can someone explain why?

$\endgroup$
1
  • 2
    $\begingroup$ In your place, I would have insisted on a solid argument of why he stated that. According to him, you cannot use therefore the embeddings as inputs to train a neural network with a single neuron and sigmoid activation. $\endgroup$ – Julio Jesus Jan 7 at 23:29
2
$\begingroup$

I think that I may understand where they are coming from. The key difference is if we're talking about inference or training.

  • Inference

There is nothing stopping you from concatenating your features. Whether they are hand-designed or a neural network's output, features can be combined however you please.

  • Training

If you don't intend to update your embeddings through training, then, there is no problem. Your embeddings will be treated as some other features and you can train your downstream model.

It gets tricky if you want to train both your downstream model and your embedding layer. Indeed, you will only be able to train your embedding layer if your full model (NN + downstream model) is differentiable. This is for instance not possible if the downstream model is a KNN model.

I assume this is what they meant when they said you couldn't concate embeddings and hand-designed features.

$\endgroup$
2
  • $\begingroup$ Your argument for the second point seems to assume that the "other" model gets tacked onto the NN and training continues, whereas I interpret the question to be taking the embedding as a list of features to be put into the "other" model, with no retraining of the NN. $\endgroup$ – Ben Reiniger Jan 21 at 22:35
  • $\begingroup$ I agree with you. I am not sure why you interpreted it this way. I explicitly say that training the full model (nn + model) may not be feasible. I updated my answer to be clearer. $\endgroup$ – Valentin Calomme Jan 21 at 23:48
1
$\begingroup$

I disagree with your interviewer, and agree with @JulioJesus's comment. Indeed, since logistic regression is a simple neural network, the interviewer's statement cannot be globally true.

But even in more distinct situations, e.g. a decision tree as the final classifier, I don't see any reason that you shouldn't include both pre-trained embedding features and additional features. Your tree might first select on some of the embedding features, and for each resulting subspace of the embedding proceed with different additional features.

Now it is true that the embedding was generated with the purpose of some classification by the original neural network, so there may or may not be a linear relationship. In that sense, perhaps the embedding features won't work optimally in the classical (esp. a linear) model. But that's a far cry from "cannot use" and "must only use with neural networks," in my opinion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.