I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take
Pandas dataframes instead of
.txt documents. I want to find the most similar sentence to a new sentence I put in from my data. However, after training, even if I give almost the same sentence that's present in the dataset, I get low-accuracy results as the top result and none of them is the sentence I modified. For example I have the sentence "This is a nice cat you have." in the dataset I train Doc2vec with, then I use the new sentence "This cat you have is quite nice." as input, and it doesn't bring up the first sentence as similar.
Data comes from an excel sheet, and has roughly the looks of:
Description | Group | Number 0 Sent: This is a sentence Regular NUM1234 1 Sent: Another sentence Regular NUM1243 2 Sent: Basically all the input Other group NUM1278 3 Sent: Creating a test case to validate the routing between applications. No action needed at this moment | Other group | NUM1287 ...etc...
I have the following code (some code not needed for comprehension was trimmed):
df = pd.read_excel("my_data.xls") df["Description"] = df["Description"].apply(lambda x: removeGeneric(x)) #removeGeneric() just strips "Sent:" from the beginning of each sentence for index, row in df.iterrows(): row["Description"] = row["Description"].lower() row["Description"] = normalize_text(row["Description"]) #normalize_text() removes stopwords defined in the nltk package and words shorter than 2 characters SentimentDocument = namedtuple('SentimentDocument', 'words tags') alldocs =  for index, row in df.iterrows(): words = gensim.utils.to_unicode(row["Description"]).split() tags = [row["Number"]] alldocs.append(SentimentDocument(words, tags)) doc_list = alldocs[:] cores = multiprocessing.cpu_count() assert gensim.models.doc2vec.FAST_VERSION > -1, "This will be painfully slow otherwise" simple_models = [ # PV-DM w/ concatenation - window=5 (both sides) approximates paper's 10-word total window size Doc2Vec(dm=1, dm_concat=1, size=100, window=5, negative=5, hs=0, min_count=2, workers=cores), # PV-DBOW Doc2Vec(dm=0, size=100, negative=5, hs=0, min_count=2, workers=cores), # PV-DM w/ average Doc2Vec(dm=1, dm_mean=1, size=100, window=10, negative=5, hs=0, min_count=2, workers=cores), ] # Speed up setup by sharing results of the 1st model's vocabulary scan simple_models.build_vocab(alldocs) # PV-DM w/ concat requires one special NULL word so it serves as template print(simple_models) for model in simple_models[1:]: model.reset_from(simple_models) print(model) models_by_name = OrderedDict((str(model), model) for model in simple_models) from random import shuffle alpha, min_alpha, passes = (0.025, 0.001, 20) alpha_delta = (alpha - min_alpha) / passes print("START %s" % datetime.datetime.now()) for epoch in range(passes): shuffle(doc_list) for name, train_model in models_by_name.items(): # Train duration = 'na' train_model.alpha, train_model.min_alpha = alpha, alpha with elapsed_timer() as elapsed: train_model.train(doc_list, total_examples=len(doc_list), epochs=1) for model in simple_models: new_sentence = "Test case creation to validation of routing between applications. No action needed" #Notice how I'm testing with a sentence very similar to one in the original dataset new_sentence = removeGeneric(new_sentence) new_sentence = normalize_text(new_sentence) print(model.docvecs.most_similar(positive=[model.infer_vector(new_sentence)],topn=2))
For this I get the following output:
[('NUM1254', 0.3154909014701843), ('NUM5247', 0.2487245500087738)] [('NUM3875', 0.20226456224918365), ('NUM3793', 0.1970052272081375)] [('NUM3585', 0.13086965680122375), ('NUM3857', 0.1298370361328125)] creating test case validate routing applications action needed moment
All the recommendations are completely unrelated, sentences like "site id plant address good owner electricity request approved number al district province" show up; the sentence it's actually close to (the sentence "Creating a test case to validate the routing between applications. No action needed at this moment" from the dataset) is not on the list.
Can you see anything that I'm doing wrong? What could I do to improve accuracy? Has anyone else experienced this inaccuracy in doc2vec's cosine similarity prediction? If I hand-code the implementation (like this for example), it does give the correct answers, which are completely different than those from doc2vec (but actually accurate).