I have found this pytorch code of transformers suitable for machine translation:
import torch import torch.nn as nn class Encoder(nn.Module): def __init__( self, src_vocab_size, device, embed_size): super(Encoder, self).__init__() self.embed_size = embed_size self.device = device self.word_embedding = nn.Embedding(src_vocab_size, embed_size) def forward(self, x): out = self.word_embedding(x) print(out) if __name__ == "__main__": device = torch.device("cuda" if torch.cuda.is_available() else "cpu") x = torch.tensor([[1, 5, 6, 4, 3, 9, 5, 2, 0], [1, 8, 7, 3, 4, 5, 6, 7, 2]]).to(device ) src_vocab_size = 10 embed_size=512 model = Encoder(src_vocab_size,device,embed_size).to(device) out = model(x)
one the first steps is embedding the word vectors, but for none word integer purposes it's possible also to use
nn.Embedding so if I want to have float numbers in x an error like this would be raised
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)
so please: tell me what alternative ways to embed floating numbers which are commonly used in non-machine-translation, non-integer tasks like most of neural nets, time series analysis, CNNs etc. I know this error should have happened, and my question is when we are dealing with integers ,someway we embed them but what is usually done equivalently when we are working with float numbers? one of possible ways is to use fully connected layer but in machine translation they also can use indexes(in form of float) and map them to another float vector ,except and avoid using word embeddings or in case of general(nn.embedding).so they probably trying to preserve some features of language so thats why they use word2vec embeddings rather than using vanilla fully connected layers, but in terms of using transformer models to get float inputs, we don't know what should be preserved, so my question is what the other researchers of this field have done?
and likewise please tell me ways to do positional embeddings for float numbers. please tell me ways to do positional embeddings, can I again (ofc its only one of ways of doing this) use fcl? I have seen
self.position_embedding = nn.Embedding(max_length, embed_size) positions = torch.arange(0, seq_length).expand(N, eq_length).to(self.device) x =self.position_embedding(positions)
but I know there is some ways using cosines, are the cosine ways are also just suitable for integers or I should go and try to find it?
by providing the code I hope that I have clarified my purpose enough avoiding misunderstanding but feel free to comment below.