My question is about the theory of this problem, and not necessarily syntax.
I'm wondering if anyone here has experience with automating HTS (Harmonized Tax Schedule) classifications, specifically training a classifier to predict the HTS code that should be assigned to a product? There are rules for these designations that are stored as descriptive text data that 'explain' the product. This seems (to me) to make predicting how a part will be classified very difficult.
For context, it would seem that NLP is heavily involved in this process, but with the high cardinality involved in these descriptions and the high amount of overlap from one description to another, are there any best practices for success in making such predictions? Similarly, I'm having a hard time finding any public research on the matter. It appears that almost all advancement in this focus was developed behind the closed doors of companies. I understand why they'd make that decision, but I am a bit surprised how little public discussion exists on the topic considering how much business it affects across the globe.
things I am considering/researching:
Word vectorizing (Word2Vec): I got the idea from another user here, but given that it is typical to train a vectorizer with a large public corpus, I am not sure how that would affect the vectoring process, as the products I am classifying are domain specific.
n-gram: I suppose using 2, 3, or 4 would be sufficient for classification, given that most descriptions that I can see are succinct. But as I said, a lot of these descriptions will have overlap, for example:
HTS CODE HTS Description ------------------ ------------------ 11111 = 'football' An inflated ball (used for kicking) 11112 = 'basketball' An inflated ball (used for dribbling) 11113 = 'ball' An inflated ball
In this case (if my understanding is correct), n-gram 6 would be needed for the first two products but only n-gram 3 is needed for the third product. Is the n-gram value something we can 'soft code' to adapt to each record or will I need to somehow find the appropriate number for n?