I'm working with the quora question pairs csv file which I loaded into a pd dataframe and isolated the qid and question so my questions are in this form :

0        What is the step by step guide to invest in sh...
1        What is the step by step guide to invest in sh...
2        What is the story of Kohinoor (Koh-i-Noor) Dia...
3        What would happen if the Indian government sto...
19408    What are the steps to solve this equation: [ma...
19409                           Is IMS noida good for BCA?
19410              How good is IMS Noida for studying BCA?

My dataset is actually bigger (500k questions) but I will use these questions to showcase my problem.

I want to identify pairs of questions that have a high probability of asking the same thing. I thought about the naive way, which is to turn each sentence into a vector using doc2vec and then for each sentence calculate the cosine similarity with every other sentence. Then, keep the one with the highest similarity and in the end print all those that have a high enough cosine similarity. The problem is this would take ages to finish so I need another approach.

Then I found an answer in another question that suggests to use clustering to solve a similar problem. So following is the code I implemented based on that answer.

"Load and transform the dataframe to a new one with only question ids and questions"
train_df = pd.read_csv("test.csv", encoding='utf-8')

questions_df.sort_values("qid", inplace=True)


# vectorization of the texts
vectorizer = TfidfVectorizer(stop_words="english")
X = vectorizer.fit_transform(questions_df['question'].values.astype('U'))
# used words (axis in our multi-dimensional space)
words = vectorizer.get_feature_names()
print("words", words)

max_iter = 300
number_of_process=2 # seads are distributed
model = KMeans(n_clusters=n_clusters, max_iter=max_iter, n_init=number_of_seeds_to_try, n_jobs=number_of_process).fit(X)

labels = model.labels_
# indices of preferable words in each cluster
ordered_words = model.cluster_centers_.argsort()[:, ::-1]

print("centers:", model.cluster_centers_)
print("labels", labels)
print("intertia:", model.inertia_)

texts_per_cluster = numpy.zeros(n_clusters)
for i_cluster in range(n_clusters):
    for label in labels:
        if label==i_cluster:
            texts_per_cluster[i_cluster] +=1

print("Top words per cluster:")
for i_cluster in range(n_clusters):
    print("Cluster:", i_cluster, "texts:", int(texts_per_cluster[i_cluster])),
    for term in ordered_words[i_cluster, :10]:


text_to_predict = "Why did Donald Trump win the elections?"
Y = vectorizer.transform([text_to_predict])
predicted_cluster = model.predict(Y)[0]

print("Cluster:", predicted_cluster, "texts:", int(texts_per_cluster[predicted_cluster])),
for term in ordered_words[predicted_cluster, :10]:

I thought that this way I could find for each sentence the cluster that it most likely belongs in and then calculate the cosine similarity between all other questions of that cluster. This way instead of doing it on all the dataset I will be doing it on far fewer documents. However using the code for an example sentence "Why did Donald Trump win the elections?" I have the following results.

Why did Donald Trump win the elections?
Cluster: 25 texts: 244

I know that my sentence belongs to cluster 25 and I can see the top words for that cluster. However how could I access the sentences that are in this cluster. Is there any way to do it?

  • $\begingroup$ Please do not cross-post duplicate questions: stackoverflow.com/q/54429050/1060350 $\endgroup$ – Has QUIT--Anony-Mousse Feb 2 '19 at 9:07
  • $\begingroup$ I'm sorry. What is the right course of action in this situation? Should I delete one of them? $\endgroup$ – thelaw Feb 2 '19 at 19:04
  • $\begingroup$ As the answer here is trivial, you can try to delete this copy (it's not likely to help anybody else). But it's mostly for the future. $\endgroup$ – Has QUIT--Anony-Mousse Feb 3 '19 at 8:31

As much as i understood from your description, you can just assign your labels as a new column to questions_df. There is no shuffling involved, so the following will do the trick:

questions_df["cluster"] = labels

| improve this answer | |
  • $\begingroup$ This will map every document to its corresponding label? $\endgroup$ – thelaw Jan 30 '19 at 12:45
  • $\begingroup$ Sure, it will. If that's what you are looking for. $\endgroup$ – Slyfest Jan 31 '19 at 7:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.