Why word2vec performs much worst than both CountVectorizer and TfidfVectorizer? [Text classification]

Question

I'm following this guide to try creating both binary classifier and multi-label classifier using MeanEmbeddingVectorizer and TfidfEmbeddingVectorizer shown in the guide above as inputs.

Both embedding vectorizers are created by first, initiating w2v from documents using gensim library, then do vector mapping to all given words in a document and vectorizes them by taking the mean of all the vectors corresponding to individual words.

After this step has completed, I tried using these embedding vectorizers as inputs into several model such as OneVsRestClassifier(SVC), RandomForestClassifier and ExtraTreesClassifier. Hence, all of the models performs worst than my expectation, with respect to what is shown in the guide.

These are the accuracies from each models (binary classifer) :

randomF_countVect: 0.8898
extraT_countVect: 0.8855
extraT_tfidf: 0.8766
randomF_tfidf: 0.8701
svc_tfidf: 0.8646
svc_countVect: 0.8604
ExtraTrees_w2v: 0.7285
ExtraTrees_w2v_tfidf: 0.7241

Multi-label classifier also produced similar result.

I'm not sure that I've done wrong.

Note that I'm working with very small documents. In which each document consists of short text (a sentence or two) and they're non-English documents. In total, whole documents only has 1163 unique words. Below is my code. Could someone please light me up?

Word2vec initiation

model = Word2Vec([my_tokenizer(item) for item in df['text']], size=100, window=5, min_count=1, workers=2)
w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}

Models

svc = Pipeline([("count_vectorizer", vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
svc_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
randomF = Pipeline([("count_vectorizer", vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
randomF_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
extraT = Pipeline([("count_vectorizer", vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
extraT_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
etree_w2v = Pipeline([("word2vec vectorizer", MeanEmbeddingVectorizer(w2v)), ("word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])
etree_w2v_tfidf = Pipeline([("tfidf word2vec vectorizer", TfidfEmbeddingVectorizer(w2v)), ("tfidf word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])


all_models = [
    ("svc", svc),
    ("svc_tfidf", svc_tfidf),
    ("randomF", randomF), 
    ("randomF_tfidf", randomF_tfidf),
    ("extraT", extraT), 
    ("extraT_tfidf", extraT_tfidf),
    ("etree_w2v", etree_w2v), 
    ("etree_w2v_tfidf", etree_w2v_tfidf)
]
scores = sorted([(name, cross_val_score(model, df['text'], df['product'], cv=kfold).mean()) 
             for name, model in all_models], 
            key=lambda args: -(args[1]))

Answer 1

My guess is that the dataset is very small. Word2Vec won't be able to capture word relationship in the embedding space with limited information. Try to train word2vec on a very large corpus to get a very good word vector before training your classifier might help.

Answer 2

I assume that you have knowledge about how things work in complex models(RandomForestClassifier, ExtraTreesClassifier), not talking much about how they work. I would explain it with reference to project which I worked on Twitter Sentiment Mining using Real-Time data Topic: Samsung Galaxy S8 Outcome: How much percent of tweets are Positive and Negative.

The possible reasons for the model to perform bad:

1. As unaki has suggested you need to try increasing the corpus size, I want to add one more point to it i.e., for any model to perform well you need to have a huge, quality corpus.

In my scenario for predicting the sentiment of tweets(real-time tweets) I used 1.5 GB of Tweets which where segregated manually as Positive or Negative. I used that for training my model and I could get an accuracy of 90%(out of 10 tweets 9 were segregated correctly). It was time consuming process but it helped me to get good outcome.

2. As you have mentioned that you have some data which is not in English, make sure that these are handled properly before giving to model i.e., if you don't want such words get rid of them or train your model to understand those other language words.

In my Scenario, I had to remove all the Non-English as my model wasn't trained to handle such words. by which my model performed well. There was an increase of 2-3% of accuracy, which was significant for me.

To Improve these Models Accuracy:

Now, to improve my model I made an new dictionary for some words like not great - which falls under a Neutral Sentiment(but I don't have one), If these words are found then that tweet is pushed to negative. By this you can expect better segregation of sentiment.

In the same way to improve you need to look for things which are not classified properly(wrongly classified by models- False Positive or False Negative these terms are with respect to Confusion Matrix) and find a pattern which is being missed by the model and use that for maximizing your models capacity.

Answer 3

Check fastText pretrained vectors (https://fasttext.cc/docs/en/crawl-vectors.html) to have a starting point generated on a bigger corpus. Then you can take these vectors and train them further with your dataset using gensim.