0
$\begingroup$

I have found this pytorch code of transformers suitable for machine translation:

import torch
import torch.nn as nn
class Encoder(nn.Module):
    def __init__(
        self,
        src_vocab_size,
        device,
        embed_size):
        super(Encoder, self).__init__()
        self.embed_size = embed_size
        self.device = device
        self.word_embedding = nn.Embedding(src_vocab_size, embed_size)
    def forward(self, x):
        out = self.word_embedding(x)
        print(out)
        
if __name__ == "__main__":
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    x = torch.tensor([[1, 5, 6, 4, 3, 9, 5, 2, 0], [1, 8, 7, 3, 4, 5, 6, 7, 2]]).to(device )
    src_vocab_size = 10
    embed_size=512
    model = Encoder(src_vocab_size,device,embed_size).to(device)
    out = model(x)

one the first steps is embedding the word vectors, but for none word integer purposes it's possible also to use nn.Embedding so if I want to have float numbers in x an error like this would be raised

RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)

so please: tell me what alternative ways to embed floating numbers which are commonly used in non-machine-translation, non-integer tasks like most of neural nets, time series analysis, CNNs etc. I know this error should have happened, and my question is when we are dealing with integers ,someway we embed them but what is usually done equivalently when we are working with float numbers? one of possible ways is to use fully connected layer but in machine translation they also can use indexes(in form of float) and map them to another float vector ,except and avoid using word embeddings or in case of general(nn.embedding).so they probably trying to preserve some features of language so thats why they use word2vec embeddings rather than using vanilla fully connected layers, but in terms of using transformer models to get float inputs, we don't know what should be preserved, so my question is what the other researchers of this field have done?

and likewise please tell me ways to do positional embeddings for float numbers. please tell me ways to do positional embeddings, can I again (ofc its only one of ways of doing this) use fcl? I have seen

self.position_embedding = nn.Embedding(max_length, embed_size)
positions = torch.arange(0, seq_length).expand(N, eq_length).to(self.device)
x =self.position_embedding(positions)

but I know there is some ways using cosines, are the cosine ways are also just suitable for integers or I should go and try to find it?

by providing the code I hope that I have clarified my purpose enough avoiding misunderstanding but feel free to comment below.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.