Categorical Variable Embedding

Question

I have a categorical variable in my labeled dataset. I trained one-hot encoded version of it in another neural network having embedding layer with a larger labeled dataset. I have obtained the weights of embedding layer. Is it possible to use embedding layer weights as a categorical variable representation like one-hot-encoding while using it in another network which has no embedding layer? For example,

One-hot-encoded variable,

  A B C D
D 0 0 0 1       
B 0 1 0 0
A 1 0 0 0
C 0 0 1 0
B 0 1 0 0
C 0 0 1 0
...

Embedding layer weights as a result of training,

   X1  X2   
A 0.2 -0.1
B 0.3 0.1
C -0.2 0.5
D 0.5 0.6

Representation of variable in the dataset using embedding layer weights above,

   X1  X2   
D 0.5 0.6
B 0.3 0.1
A 0.2 -0.1
C -0.2 0.5
B 0.3 0.1
C -0.2 0.5
...

Answer 1

Yes, it is possible but the performance of your second model will strongly depend on the performance of the first one, as well as how relevant the low-dimensional representation learned by that first model is to solve the problem you built that second model for.

You can use that first/pre-trained model up to that embedding layer, and add more layers to create that second model, which you would train with the weights already initialized with, hopefully, already useful values. That is called fine-tuning. Here is an example from Kaggle.

You could also consider freezing the weights from the first model (transfer learning) to reduce the number of parameters to learn and train your second model. This also is equivalent to creating input features from the embedding layer of the first model.

Answer 2

Yes its completely possible. Its similar to people using Word2vec embedding or any other embedding.