2
$\begingroup$

I have data which looks like following. Data is a group of sentences which are similar, but have few unique words in between like TABLEA, TABLEB etc.

java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEA
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEC
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEB
Dataframe read is null 
Dataframe read is null     
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Y  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Z  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 2652] [SQLState HY000] Operation not allowed: TABLEK is being Loaded.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 9804] [SQLState HY000] Response Row size or Constant Row size overflow.
java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 15.10.00.22] [Error 1000] [SQLState 08S01] Login failure for Connection to xxx.xx.xx.xx Tue Dec 04 02:49:47 MST 2018

Problem Statement: I want to group/cluster the data and provide a unique number to each group/cluster.

Assumptions:

  1. The groups/cluster should be formed based on the similarity. Similar sentences should be grouped in one cluster
  2. This should be unsupervised learning. If in future, some new sentence comes which is very less similar to existing cluster, it should create a new group/cluster.
  3. The sentences can be of any length
  4. The common words between sentences can appear anywhere - starting of string, in between, in the end, or so
  5. The sequence of the words matter

Output:

The outcome should be a dimension table for category like below enter image description here

Although I do get the problem statement in its abstract form, I do not know a concrete way to do this.

So far I have read about text clustering using various algorithms like cosine similarity and etc, but I am not sure if that will suffice this problem statement. One of the major problem here is, it is unsupervised. If there are any new sentences whose similarity is very less then it should create a new group.

The bigger picture goes like this

  1. Get list of all unclassified/uncategorised(I am using both interchangeably here since I am not sure which one it falls into) statements

  2. Check in the dimension table, by matching using some similarity threshold(not clear on this).

  3. If the similarity is matching above a threshold, then do nothing

  4. If similarity is less, then create a new group in the dimension table with the Description column which has common words.

I have yet to identify what is the best approach to solve this problem.Please recommend some algorithm or approach to solve this problem.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
4
$\begingroup$

For this problem, I would start with a simple bag of words model and use that as a baseline.

This is an example : https://medium.com/@MSalnikov/text-clustering-with-k-means-and-tf-idf-f099bcf95183 , http://brandonrose.org/clustering

For logs, this might be sufficient since it will easily filter out common words like date etc.

If this does not work well for a given usecase, next step is to try Doc2Vec and similar methods that project sentences to a vector space.

Improve this answer
$\endgroup$
0
$\begingroup$

The other option is to use doc2vec with regression. One exemple is to split and tag each error into words possibly cleaning up the punctuation. You train doc2vec with the whole corpus and then can infer vectors for each error. At this point you run one-vs-other regressions that will score 0 if the doc vector is not from this category and 1 if it is in this category.

Read the code of my sensimark project.

Improve this answer
$\endgroup$
0
$\begingroup$

Clustering is probably the wrong tool here. Check the assumptions...

What you really seem to be looking for is frequent subsequences.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.