# How to create clusters based on sentence similarity?

I have data which looks like following. Data is a group of sentences which are similar, but have few unique words in between like TABLEA, TABLEB etc.

java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEA
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEC
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEB
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Y  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Z  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 2652] [SQLState HY000] Operation not allowed: TABLEK is being Loaded.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 9804] [SQLState HY000] Response Row size or Constant Row size overflow.
java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 15.10.00.22] [Error 1000] [SQLState 08S01] Login failure for Connection to xxx.xx.xx.xx Tue Dec 04 02:49:47 MST 2018


Problem Statement: I want to group/cluster the data and provide a unique number to each group/cluster.

Assumptions:

1. The groups/cluster should be formed based on the similarity. Similar sentences should be grouped in one cluster
2. This should be unsupervised learning. If in future, some new sentence comes which is very less similar to existing cluster, it should create a new group/cluster.
3. The sentences can be of any length
4. The common words between sentences can appear anywhere - starting of string, in between, in the end, or so
5. The sequence of the words matter

Output:

The outcome should be a dimension table for category like below

Although I do get the problem statement in its abstract form, I do not know a concrete way to do this.

So far I have read about text clustering using various algorithms like cosine similarity and etc, but I am not sure if that will suffice this problem statement. One of the major problem here is, it is unsupervised. If there are any new sentences whose similarity is very less then it should create a new group.

The bigger picture goes like this

1. Get list of all unclassified/uncategorised(I am using both interchangeably here since I am not sure which one it falls into) statements

2. Check in the dimension table, by matching using some similarity threshold(not clear on this).

3. If the similarity is matching above a threshold, then do nothing

4. If similarity is less, then create a new group in the dimension table with the Description column which has common words.

I have yet to identify what is the best approach to solve this problem.Please recommend some algorithm or approach to solve this problem.

For this problem, I would start with a simple bag of words model and use that as a baseline.

For logs, this might be sufficient since it will easily filter out common words like date etc.

If this does not work well for a given usecase, next step is to try Doc2Vec and similar methods that project sentences to a vector space.

The other option is to use doc2vec with regression. One exemple is to split and tag each error into words possibly cleaning up the punctuation. You train doc2vec with the whole corpus and then can infer vectors for each error. At this point you run one-vs-other regressions that will score 0 if the doc vector is not from this category and 1 if it is in this category.

Read the code of my sensimark project.

Clustering is probably the wrong tool here. Check the assumptions...

What you really seem to be looking for is frequent subsequences.