1
$\begingroup$

Problem Statement:

Given the details about a product, we need to map it to its category.

Currently we are using Product Name as a feature and Product Category as the Label

There are around 50,000 categories available currently and it will grow in future.

I created a small dataset which consisted 20 categories and 100 records for each label. so the total record count is 2000. Using RandomForest I got 92% accuracy.

Problem:

So I went on to create a model with 1800 categories[labels] and records for each categories varies from 500-1500. When I ran the same model with new dataset I got only 19% accuracy and more than 50% of the predicted value pointed to the same label.

Dataset Sample:

Product_Combined    Category
2Pcs Led Light Lamp Strip Dimmer Switch Brightness Adjustable Control 12-24V 8A Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
10 Pcs 1/4" Male To 1/4" Female Screw Adapter For Tripod Camera Flash Bracket Stand Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
L-Fine A4 Tracing LED Light Pad Box(13.86x9.45 Inches) with Adjustable Light Intensity for Artists,Drawing, Sketching, Animation    Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
BZONE Solar Powered Operated Copper Wire LED Fairy Light Decorative String Lights for Indoor Outdoor Home Garden Lawn Patio Party Christmas Valentine''s Day (16.4ft, Pink Color)   Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
LitEnergy 32.5 Inch Diagonal A2 Tracing Table with LED Light and Paper  Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes

Code:

import string
import codecs
import pandas as pd
import numpy as np

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from stemming.porter2 import stem
from sklearn.metrics import confusion_matrix

from nltk.stem import PorterStemmer
from nltk.corpus import stopwords

from sklearn.model_selection import cross_val_score
from sklearn.externals import joblib

stop = stopwords.words('english')

data_file = "Book3.txt"
#Reading the input/ dataset
data = pd.read_csv( data_file, header = 0, 
    delimiter= "\t", quoting = 3, encoding = "ISO-8859-1")
data = data.dropna()
#Removing stopwords, punctuation and stemming
data['Product_Combined'] = data['Product_Combined'].apply(
    lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
data['Product_Combined'] = data['Product_Combined'].str.replace(
    '[^\w\s]',' ').replace('\s+',' ')
data['Product_Combined'] = data['Product_Combined'].apply(
    lambda x: ' '.join([stem(word) for word in x.split()]))

train_data, test_data, train_label,  test_label = train_test_split(
    data.Product_Combined, data.Breadcrumb, test_size=0.3, random_state=100)

RF = RandomForestClassifier(n_estimators=100)
vectorizer = CountVectorizer( max_features = 50000, ngram_range = ( 1,3 ) )
data_features = vectorizer.fit_transform( train_data )

RF.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data)
Output_predict = RF.predict(test_data_feature)
print ("BreadCrumb_Accuracy: " + str(np.mean(Output_predict == test_label)))

with codecs.open("out_bread_crumb.txt", "w", "utf8") as out:
    out.write("Input\tPredicted\tActual\n")
    for inp, pred, act in zip(test_data, Output_predict, test_label):
        try:
            out.write("{}\t{}\t{}\n".format(inp, pred, act))
        except:
            continue

Output:

Input   Predicted   Actual
Centuri Duster Dispos Compress Gas Duster 10 oz 2 Pk    Automotive | Exterior Accessories | Towing Products & Winches | Winches Electronics | Computers & Accessories | Computer Accessories & Peripherals | Cleaning & Repair | Compressed Air Dusters
BB Mall Phone Ring Stand Metal Stainless Steel Univers 360 Rotat Ring Kickstand iPhon 6 6s 6 s plus Samsung Note 5 Note 4 S5 iPad All SmartPhon Tablet Black    Automotive | Exterior Accessories | Towing Products & Winches | Winches Cell Phones & Accessories | Accessories | Mounts & Stands | Stands
Standard Motor Product 6444 Ignition Wire Set   Automotive | Exterior Accessories | Towing Products & Winches | Winches Automotive | Replacement Parts | Ignition Parts | Spark Plugs & Wires | Wires | Wire Sets
Walker 52271 Extension Pipe Automotive | Exterior Accessories | Towing Products & Winches | Winches Automotive | Replacement Parts | Exhaust & Emissions | Exhaust Pipes & Tips
ACDelco KS10640 Profession Time Compon Seal Automotive | Exterior Accessories | Towing Products & Winches | Winches Automotive | Replacement Parts | Bearings & Seals | Seals | Camshafts

As you can see more than 50% of the actual test data was labeled as Automotive | Exterior Accessories | Towing Products & Winches | Winches

$\endgroup$

1 Answer 1

2
$\begingroup$

There are bunch of things to take care of before you can solve the problem.

  1. How is the label distribution in the training?
  2. If the distribution is not appropriate, then you need to sample the training data appropriately.

With regards to the approach:

  1. Using random forest is appropriate.
  2. But as features to the random forest it would be better to use word vectors as input to the model. That would take into account products with same labels to have a very strong similarity score based on their names.

I have used this before almost for the exact problem and I saw a big boost in my results.

$\endgroup$
3
  • $\begingroup$ Thanks!!!. So instead of CountVectorizer using WordVect would increase the accuracy makes sense!. will Tfidf help here. If possible could u let me know how much accuracy u got when going applying the same thing. $\endgroup$ Feb 9, 2018 at 9:50
  • $\begingroup$ I dont think you would require Tfidf if you use Word Vectors. So this implies you construct a vector for a product by combining vectors for all the words of that product as one single vector of the same dimension. I would not evaluate this just on accuracy, you should definitely look at your precision, recall and f1 score. I do not remember the actual number but the evaluation metrics were good enough that I was able to use the model in production to solve the problem. $\endgroup$
    – Nischal Hp
    Feb 9, 2018 at 10:09
  • $\begingroup$ Thanks a lot!!!. Please do let me know if I need to change any other thing in my code. Also if possible any more suggestion would be very much helpful. Thanks again $\endgroup$ Feb 9, 2018 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.