Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

Question

I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the description on the purchase order is free text and people can write just about anything (often the bare minimum for the order to be approved)

Some importan considerations are:

• As the company is global, the language of the description is local (german, spanish, english, russian, chinesse, etc)

• There is no standard for the structure of the sentences

• People can input abbreviations

• For certain products, the buyers don't define if the part is a repair part or a brand new product

I have read about sklearn libraries for text analysis but I really new at machine learning algorithms (I have mostly completed basic tutorials). Most of the examples are for analyzing tweets or complains, but I don't know where to start if I want to consider variables other than text, such as quantity, unitary price, provider and other parameters; and additionally, my categories are a hierarchy of four levels based on certain rules (such as, if the unitary price is over X, then it belongs to category A)

An example of the database could be:

DESCRIPTION                             uom    Mgroup   Provider     Category   Unit    Spend     Total Quantity
CATALOG: A6-CJR-45 XRE:    N/A C-RING   FT     A        Prov1        31000000   5.1     $5.10     1
Contactor iec, 9a, 24v dc, 50/60hz (    FT3    B        Prov2        32131000   82      $164.00   2
Ducto ranurado de 2 x 3, color gris,    BAL    C        Prov1        32131000   24.34   $486.80   20
Modulo de 8 salidas aisladas 5-235 vc   ST     B        Prov2        32131000   254.74  $764.22   3
Selector no-il. plastico 2 pos. mant.   ST     B        Prov3        32131000   6.46    $32.30    5
(ELEC-L2GEW4) TERMINAL TIPO LENGÜETA P	SER    D        Prov2        39120000   3.77    $56.55    15
2 Position Selector Switch - Plastic,   M      E        Prov2        23161500   9.89    $69.23    7

So my question is, where could I start to investigate? Which algorithm would be best suited to tackle this problem?

Thank you!

Answer 1

I am working on a similar problem where we are classifying 2 million products into about 1000 categories. I converted the product descriptions using tf-idf vectorization, and then used SVM to run the supervised classification. There is a lot of optimization you can do with the sklearn package in Python for natural language processing. Additionally I struggled because I wanted to include other features beyond just text descriptions. There is a union method in sklearn for accomplishing that. The final model was about 90% accurate in classifying, but out of a 1000 classes there were quite a few with low accuracy rates from the test set. This is likely due to small training sample for those classes. So, just make sure there are enough training samples for each class when you build your model.