Please have a look at this link. This was a question I asked few months back and after some suggestions and exploring I was able to successfully use TFIDF along with MultinomialNB classifier to pretty accurately predict the 'Item' based on the
Composition column. I wrote the answer myself to tell how I solve it. But this time the same logic isn't helping really well for a similar dataset.
Old data:(referencing my previous example in the link above)
UID Item Composition 1 Water Hydrogen,Oxygen 2 Sulfuric acid Hydrogen,Sulfur,Oxygen 3 Alcohol Spirit 4 Hydrochloric acid Hydrogen,Chloride 5 Citric Acid Hydrogen,Carbon, Oxygen
New data example:
UID Item Composition 1 [Sweater] [Wool, knitting, handmade, knitting needle] 2 [Jeans] [Denim, cotton, orange thread, stonewash, blue dye] 3 [CottonTrouser] [Cotton, littlepolyster, weaving, handstitch, vcut] 4 [SilkShirt] [wormsilk, artificialsilk, weaving, hand looming, color dying, coating] 5 [Carpet] [Wool, cotton, organic cotton, knitting, sewing]
This time I have a lot of such data. There around 4200 such items in the
Item column. I am trying to use TFIDF with bigrams and trigrams and using MultinomialNB to make the algorithm learn all the items in the
Composition and predict the
1) I would like TF-IDF to so the n-grams at a word level so it can vectorise the words.
2) I would like the classifier to classify the Items based on the different compositions so it can learn what compositions put together.
For some items in
Item column, they have around 10 values that comprise in the
Composition column. So I used min_df=8 and ngram_range=(1,8), hoping it can try to build a vocabulary of around 8 words per composition.
from sklearn.feature_extraction.text import TfidfVectorizer tfidf = TfidfVectorizer(sublinear_tf=True, min_df=8, norm='l2', encoding='latin-1', ngram_range=(1,8), analyzer='word',lowercase=True,stop_words='english') features = tfidf.fit_transform(dftest.Composition.values.astype('U')).toarray() labels = dftest.Indexer features.shape
No matter what combinations of this I try the predictions are going wrong and I notice that the ngrams aren't really look at the word level in the
I need expert advice on how to better solve this problem and mistakes in my approach. Consider me a novice in this area.