Group unstructured chat logs into conversations

Question

I am new to ML/AI/NLP and am interested in tackling the following problem. I have a database of chat logs from a Discord server. The database contains the following labeled data: Author, Text, Text Created Timestamp, and Channel ID.

I think what I want to do is fairly complex, but should, in theory, be possible. I want to go through the logs, and group messages that are related to a new label, Conversation ID. In an ideal world, the model will label each piece of text with a conversation id and I can then run queries against the data to show me a related text that pertains to a specific conversation. among 1 or more individuals. (i say 1 because someone might ask a question and then reply to themselves with an answer. this will be rare)

I can have someone go through the data manually and pull out and label what constitutes a 'conversation.' for some test data. If I was able to build a successful model, I would use the conversations to look for logical groupings of questions asked with follow-up answers. That would be the ultimate goal here is to find questions/answers. I think a logical step would be first to isolate conversations.

The end goal would be to take the discussion from the beginning of time to the start of time and be able to isolate different chunks of text into a logical conversation group.

It would make sense that any conversation happens within the same Channel ID. I am not worried about cross-channel conversations. Those are rare/unlikely.

My last naive approach for this was to find every question in the corpus and then search for the subsequent N messages following that message from the same channel, then ask an LLM to identify if there was an answer given the question. I want to see if I can improve that process. My approach had a hard limit of messages after a questions where the hard limit was N, nothing is to say that there wasn't an answer to a question in N+M messages and I missed the cut-off. The idea is a model won't be limited the number of subsequent messages.

Happy to take advice from seasoned hackers on different approaches to this problem that I can go ahead and experiment with. Names of techniques, tools, models that I can research that apply specifically to this problem would be ideal.

Answer 1

"The end goal would be to take the discussion from the beginning of time to the start of time and be able to isolate different chunks of text into a logical conversation group."

I would start by clarifying this. A 'conversation' on a messaging app can be defined in a million ways. The day ended and people left the app after a sustained discussion, or the users changed the topic... but maybe came back to it later. One could argue that one conversation = one thread but the way people use threads might depend on the server.

If you don't have a definition yet, I think a conversation could be defined by some threshold that was crossed in terms of time elapsed since the last message and or change in topic. For the former, you have the data and I would look at the distribution of time between messages. It's unlikely that your distribution will be bimodal, but if it is, it might be useful.

For the latter, it will be harder but you could use embeddings (for each message) from a pre-trained model, and look at the distribution of cosine similarity between consecutive messages (or over some window) and find when that value hopefully drops, due to a topic change. You might have to filter out short messages that are very generic and build some attribution logic to assign them back to the conversation later.

Mostly speculating for this one, but maybe another way would be to use a Transformer model that can handle large chunks of text, spanning multiple messages, and look at how the attention moves as you feed the text into the model with a rolling window. I wonder if when the window is across conversations, you can see some sort of clustering.

Answer 2

This is an interesting problem to tackle, and there are several approaches you can take to solve it. Here are some suggestions:

1. Clustering Algorithms: You can use clustering algorithms like K-Means or DBSCAN to group messages into conversation threads based on their similarity. To do this you'll need to extract relevant features from the text data and use them as input to the clustering algorithm. Some common features to consider could be the TF-IDF score of the message, the length of the message, or the time difference between messages.
2. Sequence labeling: Another approach is to use sequence labeling techniques like Conditional Random Fields (CRF) or Hidden Markov Model (HMM) to label each message with a conversation ID. To do this, you'll need to train a model on your labeled data to learn the patterns that distinguish between different conversation threads. You can the use this model to label new messages automatically.
3. Transformer-based methods: With the recent advancements in dl, you can use pre-trained transformer-based models like BERT, RoBERTa, or GPT-2 to encode the text data and then use them as input to a clustering or sequence labeling algorithm. These models are powerful in understanding the context and semantics of the text and can lead to better performance.

In addition to these techniques, you can also consider pre-processing the text data to remove noise, stop words, and punctuation marks, and use stemming or lemmatization to normalize the text.

I hope this suggestions help you get start with your project :)