Similarity is the go-to way to train embeddings - use a similarity matrix (eg dot product) between the embeddings of two inputs, train to increase it for connected inputs and decrease it for inputs that are not connected.

Is it possible - and does it makes sense from a practical perspective - to train an embedding layer by connecting it to a fully connected layer (and a subsequent softmax layer) and directly predicting a target value?

Would this allow for the creation of a meaningful embedding space that could be used for further applications (eg., use it to transform data to an embedded vector as part of the input for another neural network).

For example:

class emb_pretrain_module(nn.Module):
    def __init__(self, emb_layer, output_size):
        super(emb_pretrain_module, self).__init__()
        self.emb_layer = emb_layer
        self.fc1 = nn.Linear(emb_layer.embedding_dim , output_size )
        self.fc2 = nn.Linear(output_size , output_size)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        out = self.emb_layer(x)
        out = self.fc1(out)
        out = self.fc2(out)
        out = self.softmax(out)
        return out

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.