0
$\begingroup$

I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc.

I made a training-corpus of over 700.000 lines of text. (A total of over 2 milion sentences) The korpus was thoroughly cleaned. Stopwords were removed and tokens lemmatised.

I trained gensim Word2Vec on this korpus and validated the model aginst a test dataset. The dataset consists of about 500 tickets that are in maybe 40 different clusters. Now after many tries the best my model can achieve is about 20% hits and 65% false positivs. (The word similarities however, are very acurate. Using those wordvectors in document-classification is when things get inacurate)

I tried the model on live data and again, it finds similar tickets and clusters them, but it mixes in a lot of noise and probably misses a lot of tickets.

For the document vektor i use the following code:

for row in data.index:
        doc = nlp(data.at[row, 'Text'])
        tokens = ' '.join(token.lemma_.lower() for token in doc if token.pos_ in (
            'NOUN',
            'PROPN',
            'VERB',
            'ADV',
            'ADJ'
        )and not token.is_stop and token.lower_ not in wordlist)
        data.at[row, 'Tokens'] = tokens
data = data[data.Tokens != '']
data.loc[:, ('Vektoren')] = data['Tokens'].map(lambda s: nlp(s).vector)

One idea i have is to train a NER Model first, filter out all Entitys and build a second vector from those. And then combine the NER-Vector and Text-Vector with different weights.

I would be very thankfull for some advice on how to achieve better accuracy

Oh and theese are the final training parameters:

model = gensim.models.Word2Vec(token_lists,
                               min_count=5,
                               sg=0,
                               cbow_mean=0,
                               vector_size=300,
                               sample=0.00001,
                               negative=5,
                               ns_exponent=0.75,
                               window=20,
                               alpha=0.025,
                               min_alpha=0.0001,
                               workers=cores - 1,
                               callbacks=[loss_logger],
                               compute_loss=True,
                               epochs=10)
return model
Improve this question
$\endgroup$
2
  • $\begingroup$ Please explain a bit better how you get the document vector. Unfortunately, your code is not self-explanatory: you are using some attributes and functions (e.g. nlp) that are not documented, here. $\endgroup$
    – Broele
    Apr 5 at 18:23
  • $\begingroup$ Sorry! I use spacy for tokenisation. The nlp object creates the pipeline to access tokens, pos-tags and vectors. So i use nlp to extract relevant tokens first and add them to a list. In the final line i access the list of tokens and use spacy on them again to generate a vector-representation of the text. Sapcy does this be generating a mean vector of all wordvectors from the token list $\endgroup$
    – Roland
    Apr 6 at 9:23

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.