How to correctly infer vectors in Gensim doc2vec?

Question

I would like to know which is the correct procedure for inferring vectors in Gensim doc2vec.

I have a dataframe df with a feature, called name, and composed of two subsets train and test.

df = train + test

My aim is to find the most similar name in train given a name in test. For doing this I have to train the doc2vec model, and I have two possible choices:

• train the model on the entire df and then infer the most similar name by model.infer_vector() on test.
• train the model on train, letting out test, and then use model.infer_vector() on test.

I suppose that the correct procedure is first one, but I am not sure.

Also, so doing, there is the possibility that the most similar name given test will be again in test and not in train.

Answer 1

I would use the first approach, given that both train and test are known, there is no need of generalization i.e. you don't expect unseen vectors.

In order to avoid the problem you mentioned, you have to find the most similar vector to a vector in test considering only vectors in train. For example:

train = [v1, v2, v3]
test = [v4, v5]
most_similar = {}
for vector in test:
most_similar[vector] = v1
  for vector2 in train:
   if similarity(vector2, vector) > similarity(most_similar[vector], vector):
     most_similar[vector] = vector2

In the end in most_similar you have the most similar vector in train for each vector in test.