I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc.
I made a training-corpus of over 700.000 lines of text. (A total of over 2 milion sentences) The korpus was thoroughly cleaned. Stopwords were removed and tokens lemmatised.
I trained gensim Word2Vec on this korpus and validated the model aginst a test dataset. The dataset consists of about 500 tickets that are in maybe 40 different clusters. Now after many tries the best my model can achieve is about 20% hits and 65% false positivs. (The word similarities however, are very acurate. Using those wordvectors in document-classification is when things get inacurate)
I tried the model on live data and again, it finds similar tickets and clusters them, but it mixes in a lot of noise and probably misses a lot of tickets.
For the document vektor i use the following code:
for row in data.index: doc = nlp(data.at[row, 'Text']) tokens = ' '.join(token.lemma_.lower() for token in doc if token.pos_ in ( 'NOUN', 'PROPN', 'VERB', 'ADV', 'ADJ' )and not token.is_stop and token.lower_ not in wordlist) data.at[row, 'Tokens'] = tokens data = data[data.Tokens != ''] data.loc[:, ('Vektoren')] = data['Tokens'].map(lambda s: nlp(s).vector)
One idea i have is to train a NER Model first, filter out all Entitys and build a second vector from those. And then combine the NER-Vector and Text-Vector with different weights.
I would be very thankfull for some advice on how to achieve better accuracy
Oh and theese are the final training parameters:
model = gensim.models.Word2Vec(token_lists, min_count=5, sg=0, cbow_mean=0, vector_size=300, sample=0.00001, negative=5, ns_exponent=0.75, window=20, alpha=0.025, min_alpha=0.0001, workers=cores - 1, callbacks=[loss_logger], compute_loss=True, epochs=10) return model