I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc.

I made a training-corpus of over 700.000 lines of text. (A total of over 2 milion sentences) The korpus was thoroughly cleaned. Stopwords were removed and tokens lemmatised.

I trained gensim Word2Vec on this korpus and validated the model aginst a test dataset. The dataset consists of about 500 tickets that are in maybe 40 different clusters. Now after many tries the best my model can achieve is about 20% hits and 65% false positivs. (The word similarities however, are very acurate. Using those wordvectors in document-classification is when things get inacurate)

I tried the model on live data and again, it finds similar tickets and clusters them, but it mixes in a lot of noise and probably misses a lot of tickets.

For the document vektor i use the following code:

for row in data.index:
        doc = nlp(data.at[row, 'Text'])
        tokens = ' '.join(token.lemma_.lower() for token in doc if token.pos_ in (
        )and not token.is_stop and token.lower_ not in wordlist)
        data.at[row, 'Tokens'] = tokens
data = data[data.Tokens != '']
data.loc[:, ('Vektoren')] = data['Tokens'].map(lambda s: nlp(s).vector)

One idea i have is to train a NER Model first, filter out all Entitys and build a second vector from those. And then combine the NER-Vector and Text-Vector with different weights.

I would be very thankfull for some advice on how to achieve better accuracy

Oh and theese are the final training parameters:

model = gensim.models.Word2Vec(token_lists,
                               workers=cores - 1,
return model
  • $\begingroup$ Please explain a bit better how you get the document vector. Unfortunately, your code is not self-explanatory: you are using some attributes and functions (e.g. nlp) that are not documented, here. $\endgroup$
    – Broele
    Commented Apr 5, 2023 at 18:23
  • $\begingroup$ Sorry! I use spacy for tokenisation. The nlp object creates the pipeline to access tokens, pos-tags and vectors. So i use nlp to extract relevant tokens first and add them to a list. In the final line i access the list of tokens and use spacy on them again to generate a vector-representation of the text. Sapcy does this be generating a mean vector of all wordvectors from the token list $\endgroup$
    – Roland
    Commented Apr 6, 2023 at 9:23


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.