# (Pre)training an Embedding Layer via prediction (as an alternative to similarity) - does that makes sense?

Similarity is the go-to way to train embeddings - use a similarity matrix (eg dot product) between the embeddings of two inputs, train to increase it for connected inputs and decrease it for inputs that are not connected.

Is it possible - and does it makes sense from a practical perspective - to train an embedding layer by connecting it to a fully connected layer (and a subsequent softmax layer) and directly predicting a target value?

Would this allow for the creation of a meaningful embedding space that could be used for further applications (eg., use it to transform data to an embedded vector as part of the input for another neural network).

For example:

class emb_pretrain_module(nn.Module):
def __init__(self, emb_layer, output_size):
super(emb_pretrain_module, self).__init__()

self.emb_layer = emb_layer
self.fc1 = nn.Linear(emb_layer.embedding_dim , output_size )
self.fc2 = nn.Linear(output_size , output_size)
self.softmax = nn.Softmax(dim=1)

def forward(self, x):
out = self.emb_layer(x)
out = self.fc1(out)
out = self.fc2(out)
out = self.softmax(out)
return out