Skip to main content
2 votes

Why do semantically different words produce similar embeddings?

I explored this very problem in one of my medium posts.Why are Cosine similarities almost always positive. Quoting from that: In other words, the cosine similarity has a positive contribution if the ...
Vaibhav Garg's user avatar
2 votes

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

Take a look at the ALiBi paper: https://arxiv.org/abs/2108.12409 For me, the takeaways were: The sin/cos idea in the "Attention is All You Need" added complexity in the hope it would ...
Darren Cook's user avatar
  • 1,074
2 votes
Accepted

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I found this post really helpful for understanding some of the nice properties behind positional embeddings. I'll give a short summary of the relevant portions of the post in my answer, but I highly ...
Alexander Wan's user avatar
2 votes
Accepted

A question about contextual embeddings in the decoder only transformer architecture (gpt)

The second subscript refers to the position within the sequence of the contextual embedding being computed. The inner subscript range refers to what input tokens are usted as input for the previously ...
noe's user avatar
  • 27k
2 votes
Accepted

How to combine two vector embeddings into one?

How do you plan on using these embeddings? You can definitely use concatenated embeddings for similarity/retrieval, but only when comparing concat embeddings to other concat embeddings. Your point ...
Karl's user avatar
  • 756
1 vote
Accepted

How OpenAI embeddings work?

OpenAI described their embeddings in this article. In the last version, they also incorporate the matryoshka representation learning technique. The answers to your questions: The corpus used to train ...
noe's user avatar
  • 27k
1 vote
Accepted

How can I use contextual embeddings with BERT for sentiment analysis/classification

For this kind of setup, you should use the output at the first position and train a linear classifier over your 3 labels. BERT was trained with inputs that were prepended a special token ...
noe's user avatar
  • 27k
1 vote
Accepted

How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder

First, note that the purpose of next sentence prediction objective is not to contribute to the contextual embeddings part, but to allow other downstream tasks like sentence classification and textual ...
noe's user avatar
  • 27k
1 vote
Accepted

What does maximize average log probability mean?

This formula does not mazimize anything. You have to maximize it, usually with gradient descent.
noe's user avatar
  • 27k
1 vote

How to use location information as feature?

So embeddings for cities can be either loaded from pre-existing models, or created. Some of the most popular embeddings I'm familiar with include Glove and Word2Vec. The easiest way is probably to use ...
brewmaster321's user avatar
1 vote

Appropriate input size for nn.Embedding

nn.embedding layer is a sytactic sugar equivalent to one hot vector+ linear layer. Suppose you have 2 distinct variables. and you want your model to learn their ...
lateBloomer's user avatar
1 vote

Training embeddings on own dataset

Depending on the architecture of your RAG system, there can in some cases be efficiencies in using the same embeddings as your LLM of choice, but this is not a hard requirement. For example, if you ...
brewmaster321's user avatar
1 vote

Recommended way to embed a text thousands of tokens long?

One way is to just take the embedding of the CLS token. Another way is to average all the tokens. Key search terms are sentence embeddings and document embeddings. The best models are specifically ...
Darren Cook's user avatar
  • 1,074
1 vote

building embeddings for Phrases from scratch

If you want to train on phrases, then you will have to devise your tokenizer that way. Pseudo code will look like: ...
lateBloomer's user avatar
1 vote

building embeddings for Phrases from scratch

There are several ways word-embeddings are trained, however most of them require a ton of data. They usually involve learning vector representations that are useful for some self-supervised objective, ...
Alexander Wan's user avatar
1 vote
Accepted

word embeddings, what are contextual word embeddings

Contextual word embeddings are a type of word representation that captures not only the semantic meaning of individual words, but also the context in which they appear. This is completely opposite to ...
Harshad Patil's user avatar
1 vote

Semantic search - combine text and image embedding

Currently I am working in a marketplace too, and I am trying to combine text and embedding features. I am doing this by simply concatenation of them. It's important to normalize the text features and ...
Фёдор Курушин's user avatar
1 vote

How to retrain Glove Vectors on top of my own data?

If you would like to re-train your specific dataset with a pre-trained glove model you could try this code: ...
Sayidina ahmadal qososyi Qosyi's user avatar
1 vote
Accepted

How is determined the context's dimension in Doc2Vec?

The dimension of the context in Gensim's Doc2Vec is determined by the parameter vector_size when you initialize the Doc2Vec model. This parameter sets the ...
Harshad Patil's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible