I'm following this guide to try creating both binary classifier and multi-label classifier using MeanEmbeddingVectorizer and TfidfEmbeddingVectorizer shown in the guide above as inputs.
Both embedding vectorizers are created by first, initiating w2v from documents using gensim library, then do vector mapping to all given words in a document and vectorizes them by taking the mean of all the vectors corresponding to individual words.
After this step has completed, I tried using these embedding vectorizers as inputs into several model such as OneVsRestClassifier(SVC), RandomForestClassifier and ExtraTreesClassifier. Hence, all of the models performs worst than my expectation, with respect to what is shown in the guide.
These are the accuracies from each models (binary classifer) :
randomF_countVect: 0.8898
extraT_countVect: 0.8855
extraT_tfidf: 0.8766
randomF_tfidf: 0.8701
svc_tfidf: 0.8646
svc_countVect: 0.8604
ExtraTrees_w2v: 0.7285
ExtraTrees_w2v_tfidf: 0.7241
Multi-label classifier also produced similar result.
I'm not sure that I've done wrong.
Note that I'm working with very small documents. In which each document consists of short text (a sentence or two) and they're non-English documents. In total, whole documents only has 1163 unique words. Below is my code. Could someone please light me up?
Word2vec initiation
model = Word2Vec([my_tokenizer(item) for item in df['text']], size=100, window=5, min_count=1, workers=2)
w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}
Models
svc = Pipeline([("count_vectorizer", vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
svc_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
randomF = Pipeline([("count_vectorizer", vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
randomF_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
extraT = Pipeline([("count_vectorizer", vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
extraT_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
etree_w2v = Pipeline([("word2vec vectorizer", MeanEmbeddingVectorizer(w2v)), ("word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])
etree_w2v_tfidf = Pipeline([("tfidf word2vec vectorizer", TfidfEmbeddingVectorizer(w2v)), ("tfidf word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])
all_models = [
("svc", svc),
("svc_tfidf", svc_tfidf),
("randomF", randomF),
("randomF_tfidf", randomF_tfidf),
("extraT", extraT),
("extraT_tfidf", extraT_tfidf),
("etree_w2v", etree_w2v),
("etree_w2v_tfidf", etree_w2v_tfidf)
]
scores = sorted([(name, cross_val_score(model, df['text'], df['product'], cv=kfold).mean())
for name, model in all_models],
key=lambda args: -(args[1]))