Random Forest Multiclass Classification

Question

Problem Statement:

Given the details about a product, we need to map it to its category.

Currently we are using Product Name as a feature and Product Category as the Label

There are around 50,000 categories available currently and it will grow in future.

I created a small dataset which consisted 20 categories and 100 records for each label. so the total record count is 2000. Using RandomForest I got 92% accuracy.

Problem:

So I went on to create a model with 1800 categories[labels] and records for each categories varies from 500-1500. When I ran the same model with new dataset I got only 19% accuracy and more than 50% of the predicted value pointed to the same label.

Dataset Sample:

Product_Combined    Category
2Pcs Led Light Lamp Strip Dimmer Switch Brightness Adjustable Control 12-24V 8A Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
10 Pcs 1/4" Male To 1/4" Female Screw Adapter For Tripod Camera Flash Bracket Stand Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
L-Fine A4 Tracing LED Light Pad Box(13.86x9.45 Inches) with Adjustable Light Intensity for Artists,Drawing, Sketching, Animation    Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
BZONE Solar Powered Operated Copper Wire LED Fairy Light Decorative String Lights for Indoor Outdoor Home Garden Lawn Patio Party Christmas Valentine''s Day (16.4ft, Pink Color)   Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes
LitEnergy 32.5 Inch Diagonal A2 Tracing Table with LED Light and Paper  Arts, Crafts & Sewing | Painting, Drawing & Art Supplies | Drawing | Light Boxes

Code:

import string
import codecs
import pandas as pd
import numpy as np

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from stemming.porter2 import stem
from sklearn.metrics import confusion_matrix

from nltk.stem import PorterStemmer
from nltk.corpus import stopwords

from sklearn.model_selection import cross_val_score
from sklearn.externals import joblib

stop = stopwords.words('english')

data_file = "Book3.txt"
#Reading the input/ dataset
data = pd.read_csv( data_file, header = 0, 
    delimiter= "\t", quoting = 3, encoding = "ISO-8859-1")
data = data.dropna()
#Removing stopwords, punctuation and stemming
data['Product_Combined'] = data['Product_Combined'].apply(
    lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
data['Product_Combined'] = data['Product_Combined'].str.replace(
    '[^\w\s]',' ').replace('\s+',' ')
data['Product_Combined'] = data['Product_Combined'].apply(
    lambda x: ' '.join([stem(word) for word in x.split()]))

train_data, test_data, train_label,  test_label = train_test_split(
    data.Product_Combined, data.Breadcrumb, test_size=0.3, random_state=100)

RF = RandomForestClassifier(n_estimators=100)
vectorizer = CountVectorizer( max_features = 50000, ngram_range = ( 1,3 ) )
data_features = vectorizer.fit_transform( train_data )

RF.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data)
Output_predict = RF.predict(test_data_feature)
print ("BreadCrumb_Accuracy: " + str(np.mean(Output_predict == test_label)))

with codecs.open("out_bread_crumb.txt", "w", "utf8") as out:
    out.write("Input\tPredicted\tActual\n")
    for inp, pred, act in zip(test_data, Output_predict, test_label):
        try:
            out.write("{}\t{}\t{}\n".format(inp, pred, act))
        except:
            continue

Output:

Input   Predicted   Actual
Centuri Duster Dispos Compress Gas Duster 10 oz 2 Pk    Automotive | Exterior Accessories | Towing Products & Winches | Winches Electronics | Computers & Accessories | Computer Accessories & Peripherals | Cleaning & Repair | Compressed Air Dusters
BB Mall Phone Ring Stand Metal Stainless Steel Univers 360 Rotat Ring Kickstand iPhon 6 6s 6 s plus Samsung Note 5 Note 4 S5 iPad All SmartPhon Tablet Black    Automotive | Exterior Accessories | Towing Products & Winches | Winches Cell Phones & Accessories | Accessories | Mounts & Stands | Stands
Standard Motor Product 6444 Ignition Wire Set   Automotive | Exterior Accessories | Towing Products & Winches | Winches Automotive | Replacement Parts | Ignition Parts | Spark Plugs & Wires | Wires | Wire Sets
Walker 52271 Extension Pipe Automotive | Exterior Accessories | Towing Products & Winches | Winches Automotive | Replacement Parts | Exhaust & Emissions | Exhaust Pipes & Tips
ACDelco KS10640 Profession Time Compon Seal Automotive | Exterior Accessories | Towing Products & Winches | Winches Automotive | Replacement Parts | Bearings & Seals | Seals | Camshafts

As you can see more than 50% of the actual test data was labeled as Automotive | Exterior Accessories | Towing Products & Winches | Winches

Answer 1

There are bunch of things to take care of before you can solve the problem.

1. How is the label distribution in the training?
2. If the distribution is not appropriate, then you need to sample the training data appropriately.

With regards to the approach:

1. Using random forest is appropriate.
2. But as features to the random forest it would be better to use word vectors as input to the model. That would take into account products with same labels to have a very strong similarity score based on their names.

I have used this before almost for the exact problem and I saw a big boost in my results.