Clusterise a group of answers taking the question as relevant information

Question

I am solving a problem where I group answers to a given question into clusters using k-means algorithm. The steps I follow are:

1. For every answer I get the corresponding vector. Reduce the vector dimension
2. I pass the vectors into sckitlearn kmeans implementation and get clusters as output

Now, as a new requirement I want to include the question as extra context on the clustering. Before this, I was completely ignoring it. For example given the question: What would you take to a picnic? The answers beer and soft drink could be on the same cluster because they are beverages, but on the question: What can you buy for kid anniversary party? They shouldn't be.

So my problem would be, how to modify my algorithm so that the formed clusters are question relevant? Or in other words, how to include information about the question in the data so that the algorithm can create clusters according to this.

I've tried some ideas like vectorising also the question and adding it to every answer vectore to have a new resultant vector on step 1, but it doesn't seem to make a difference.

EDIT: Example

Question: "What would you do during a picnic?"
Answers: "Eat burguer","Drink a beer","Play football","play soccer", "Enjoy a root beer","Swallow a steak","Eat a salad", "Something else"
Clusters that could be formed: ["Eat burguer","Swallow a steak"], ["Drink a beer",  "Enjoy a root beer"], ["Play football","play soccer"]

Answer 1

Below there is a working example based on the approach I just described in my comment:

# pip install transformers
from sklearn.cluster import KMeans
from transformers import BertTokenizer, BertModel
import torch
import numpy as np
import pandas as pd

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")


# Example question and answers
question = "What would you do during a picnic?"
answers = [
    "Eat burger", "Drink a beer", "Play football", "Play soccer",
    "Enjoy a root beer", "Swallow a steak", "Eat a salad", "Something else"
]

# Load pre-trained BERT model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

# Tokenize and generate embeddings for the question
question_tokens = tokenizer(question, return_tensors='pt', padding='max_length', max_length=128, truncation=True)
with torch.no_grad():
    question_embedding = model(**question_tokens).last_hidden_state.mean(dim=1).numpy()

# Tokenize and generate embeddings for answers and concatenate with question embedding
answer_embeddings = []
for answer in answers:
    answer_tokens = tokenizer(answer, return_tensors='pt', padding='max_length', max_length=128, truncation=True)
    with torch.no_grad():
        answer_embedding = model(**answer_tokens).last_hidden_state.mean(dim=1).numpy()
    joint_embedding = np.concatenate((question_embedding, answer_embedding), axis=1)
    answer_embeddings.append(joint_embedding)

# Perform k-means clustering
k = 3  # Number of clusters
kmeans = KMeans(n_clusters=k)
X = np.array(answer_embeddings).reshape(len(answers), -1)
cluster_labels = kmeans.fit_predict(X)

# Print clustering results
for i, answer in enumerate(answers):
    print(f"Answer: {answer} | Cluster: {cluster_labels[i]}")


df =pd.DataFrame(X, columns = [f"dim_{x}" for x in range(X.shape[1])])
pca= PCA(n_components=2).fit(df)
X2D = pca.transform(df)


colors =["red","blue","black"]
for i, answer in enumerate(answers):
    plt.scatter(X2D[i][0], X2D[i][0], color=colors[cluster_labels[i]], label=f'Cluster {cluster_labels[i]}')
    plt.text(X2D[i][0] + i*.1, X2D[i][0] + i*.1, answer, fontsize=8, ha='right', va='bottom')
plt.title(f"{question}");

Outputs:

Hope it helps!

Answer 2

Here are a few ideas on how to modify your algorithm so that the formed clusters are question relevant:

1.Use a weighted K-means algorithm - In a weighted K-means algorithm, each data point is assigned a weight. The weight of a data point determines how much it influences the centroids of the clusters. You can use the question vector to weight the answer vectors. This will help the algorithm to pay more attention to the question when it is clustering the answers.

2.Use a semi-supervised K-means algorithm - In a semi-supervised K-means algorithm, some of the data points are labeled, while the others are unlabeled. The labeled data points are used to train the algorithm, and the unlabeled data points are used to evaluate the algorithm. You can use the question vector to label the answer vectors. This will help the algorithm to learn the relationship between the questions and the answers.

3.Use a hierarchical clustering algorithm - In a hierarchical clustering algorithm, the data points are recursively clustered together. The clusters are formed based on the similarity between the data points. You can use the question vector to guide the hierarchical clustering algorithm. This will help the algorithm to create clusters that are question relevant.

I hope these ideas help you to modify your algorithm so that the formed clusters are question relevant. Let me know if you have any other questions.

In addition to the above, here are some other things you can try:

Use a different vectorization method for the question and answer vectors. For example, you could use a bag-of-words vectorizer or a TF-IDF vectorizer. Try using a different number of clusters. Try using a different distance metric. Try using a different K-means implementation. Experiment with different parameters and see what works best for your data.

Answer 3

One approach is semi-supervised clustering which can use anchor words as seeds to guide the formation of clusters. Correlation Explanation (CorEx) is a Python package for this style of clustering.