0
$\begingroup$

I have a multi-class text classification problem in hand this is similar to product category mapping where we map products to its correct Category based on the text content provided.

I first created a solution with Hashing Vector and SGD classifier with actually gave around ~84% accuracy.

After going through many online content I found that Doc2Vec is the current cutting-edge representation of Document or paragraph in numerical format.

So I changed my solution to use Doc2Vec method for Feature Engineering but the accuracy got from this is only ~54%.

Code:

Reading and Cleansing Data

import logging
import datetime
import re
import string
import codecs
import pandas as pd
import numpy as np

from gensim.models.doc2vec import Doc2Vec,TaggedDocument
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.utils.class_weight import compute_class_weight
from sklearn.metrics import accuracy_score
from sklearn.linear_model import SGDClassifier

import warnings
warnings.filterwarnings("ignore")

#Reading the input/ dataset
data_file = "Consolidated_input_dataset.txt"
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

#Cleansing the input dataset removing non alphabets 
data['cleansed_desc'] = data.COMMODITY_DESC.str.lower().str.replace('[^a-z]',' ').str.replace('\s+',' ')

#Spliting to list for traing Doc2Vec
data['cleansed_desc_split'] = data.cleansed_desc.str.split()
train_data, test_data, train_label,  test_label = train_test_split(data[["cleansed_desc", "cleansed_desc_split"]], 
                                                                   data[["Label"]], 
                                                                   test_size=0.3, random_state=100, stratify=data.Label)

Hashing Vector:

sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))


#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

vectorizer = HashingVectorizer(n_features=90000, ngram_range=(1,3))

vectorizer.fit(train_data.cleansed_desc)
data_features = vectorizer.transform(train_data.cleansed_desc)
sgd_model.fit(data_features, train_label.Label)

#Predict Output
test_features = vectorizer.transform(test_data.cleansed_desc)
output_node1_predict = sgd_model.predict(test_features)

print(accuracy_score(test_label.Label, output_node1_predict))

Output:

84%

Doc2Vec:

#Creating Doc2Vec

data_tagged = train_data.apply(
    lambda r: TaggedDocument(words=r['cleansed_desc_split'], tags=[train_label.loc[r.name].Label]), axis=1)

doc2vec_test = Doc2Vec(dm=0, vector_size=100, negative=5, hs=0, min_count=2, sample=0, epochs=5, workers=8)
doc2vec_test.build_vocab(data_tagged)
doc2vec_test.train(data_tagged, total_examples=doc2vec_test.corpus_count, epochs=doc2vec_test.iter)

train_data["Doc2Vec"] = train_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))
test_data["Doc2Vec"] = test_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))

#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))

output:

54%

Parameters for Doc2 Vec:

Vector Size =100
Window = 10
Epoch=5
min_count=2
Negative=5

Total number of Documents = 3450000+
Vocabulary Size = 46000+
$\endgroup$
1
$\begingroup$

If I read your model correctly, you only performed 5 epochs with the Doc2Vec model. This is probably not be enough for the network to learn the word embeddings. Has your loss leveled out after 5 epochs? Try running it for 50 epochs and see if it makes any difference. Conceivably, you could need thousands of epochs to achieve a reasonable model.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have updated the epochs to 30 and have initiated another run let see, will mark your answer once I evaluate the results. but in the other hand do we really need 1000s epochs for the variable to converge. $\endgroup$ – The6thSense Nov 29 '18 at 3:49
  • 1
    $\begingroup$ How deep is the ocean? While each model is different, it’s not uncommon for NNets to require millions of epochs to converge. They are a two edged sword. $\endgroup$ – Skiddles Nov 29 '18 at 4:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.