Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
An interviewer told me that we cannot concatenate an embedding from a neural network (such as a pre-trained image representation) and hand designed features (such as image metadata) for use in a linear model such as logistic regression. He says they must only be used with neural networks.
I understand the resulting data may not be linearly separable but that to me seems like a case-by-case empirical question.
Is he right? Can someone explain why?
I think that I may understand where they are coming from. The key difference is if we're talking about inference or training.
There is nothing stopping you from concatenating your features. Whether they are hand-designed or a neural network's output, features can be combined however you please.
If you don't intend to update your embeddings through training, then, there is no problem. Your embeddings will be treated as some other features and you can train your downstream model.
It gets tricky if you want to train both your downstream model and your embedding layer. Indeed, you will only be able to train your embedding layer if your full model (NN + downstream model) is differentiable. This is for instance not possible if the downstream model is a KNN model.
I assume this is what they meant when they said you couldn't concate embeddings and hand-designed features.
I disagree with your interviewer, and agree with @JulioJesus's comment. Indeed, since logistic regression is a simple neural network, the interviewer's statement cannot be globally true.
But even in more distinct situations, e.g. a decision tree as the final classifier, I don't see any reason that you shouldn't include both pre-trained embedding features and additional features. Your tree might first select on some of the embedding features, and for each resulting subspace of the embedding proceed with different additional features.
Now it is true that the embedding was generated with the purpose of some classification by the original neural network, so there may or may not be a linear relationship. In that sense, perhaps the embedding features won't work optimally in the classical (esp. a linear) model. But that's a far cry from "cannot use" and "must only use with neural networks," in my opinion.
