0
$\begingroup$

Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the average of all vectors of words referring to animals (like family and species names and common words like "cat").

Would one say this averaged vector represents the "prototypical animal"?

Would one expect this vector to be similar to the vectors of "animal" and "animal-like"?

How would the averaged vector probably look like? Will for example many dimensions be cancelled out by averaging, and only a small number of non-vanishing dimensions remain? Will this characteristic "spectrum" be said to represent "animal-likeness"? Or won't there be something characteristic to be found?

(I could try to find out by myself by using available word embeddings, but I have no idea how to extract the words referring to animals from some several 100,000 words.)

Improve this question
$\endgroup$
1
  • $\begingroup$ You can also add the NLP tag so more experts see this question. $\endgroup$
    – Memristor
    Jun 13 at 11:07

1 Answer 1

Reset to default
1
$\begingroup$

Many questions on a single post, let's go step by step:

Assume you have taken the average of all vectors of words referring to animals (like family and species names, common terms like "cat"). Would one say this averaged vector represents the "prototypical animal"?

For a given dataset (of documents) you can refer to this vector as the average of animal vectors. If this vector can be understood as prototypical animal depends on the dataset, the dimensions chosen for the embedding, the type of embedding, the vectors chosen for making the average, etc. For example if you are training an embedding with two dimensions, and your corpus (set of documents) consists of two classes of documents (one dealing with football terms and the other with animal terms), it is quite probable that one of the dimensions of the resultant embedding vectors will refer to animal terms.

Would one expect this vector to be similar to the vectors of "animal" and "animal-like"?

Again it depends on the dataset, dimensions chosen, vectors chosen, weights for those vectors while making the average, etc. You are not guaranteed a 100% coincidence.

How would the averaged vector probably look like? Will for example many dimensions be cancelled out by averaging, and only a small number of non-vanishing dimensions remain?

Let's assume your corpus is only dealing with animal terms (e.g. veterinary documents), let's assume you chose 4 dimensions for the embedding space: in order to have cancellation of dimensions (i guess you mean average is 0) your embedding training must have discovered vectors with opposite values for some dimensions, e.g. it may have learned that mammals are the opposite of reptiles (or directly non-mammalian), then assume you have those two vectors normalized and summed, the value for one of the dimensions may cancel; again it doesn't have to be exact.

Will this characteristic "spectrum" be said to represent "animal-likeness"? Or won't there be something characteristic to be found?

With embedding vectors you can find similar vectors, so yes you can check for animal-likeness for each term, again it depend on your dataset; what if the word four appears always near the word dog? Clearly the training process will assume that four has some animal-likeness but we know the word as such is not related to any animal, plus you can have two legs animals (e.g. birds).

I'd recommend you this post towardsdatascience: The Simple Approach to Word Embedding for Natural Language Processing using Python, it use a custom corpus and obtains the corresponding embedding set. Remember those are not contextual embedding vectors (like those used in Transformers) so each word can have at most an embedding vector.

To clarify the difference between word embeddings and contextual embeddings: stackoverflow: differences between contextual embedding and word embedding

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks! I should have been more specific in my question: I'm thinking of really large corpora of documents which are thematically open - like the training data for GPT. And I am thinking of vector dimensions of at least some hundreds, not of two or four. Should I clarify this in my question? $\endgroup$
    – Hans-Peter Stricker
    Jun 13 at 11:33
  • $\begingroup$ Maybe, the thing is your link was specific to Word2Vec so that's why I centered my answer on this. Word2Vec embedding are not contextual embeddings: stackoverflow.com/questions/62272056/… $\endgroup$
    – Memristor
    Jun 13 at 11:36
  • $\begingroup$ But what did my question have to do with the difference between word and contextual embeddings? $\endgroup$
    – Hans-Peter Stricker
    Jun 13 at 11:38
  • $\begingroup$ You put a link to word embeddings but says that you were talking about contextual embeddings, since I have wasted my time on answering you, I decided to give you a link to understand the difference. $\endgroup$
    – Memristor
    Jun 13 at 11:48
  • $\begingroup$ I would have said this unintentionally. Why did you think I was talking about contextual embeddings? $\endgroup$
    – Hans-Peter Stricker
    Jun 13 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.