Please have a look at this link. This was a question I asked few months back and after some suggestions and exploring I was able to successfully use TFIDF along with MultinomialNB classifier to pretty accurately predict the 'Item' based on the Composition column. I wrote the answer myself to tell how I solve it. But this time the same logic isn't helping really well for a similar dataset.

Old data:(referencing my previous example in the link above)

UID    Item               Composition
1      Water              Hydrogen,Oxygen
2      Sulfuric acid      Hydrogen,Sulfur,Oxygen
3      Alcohol            Spirit
4      Hydrochloric acid  Hydrogen,Chloride
5      Citric Acid        Hydrogen,Carbon, Oxygen

New data example:

UID    Item                 Composition
1      [Sweater]            [Wool, knitting, handmade, knitting needle]
2      [Jeans]              [Denim, cotton, orange thread, stonewash, blue dye]
3      [CottonTrouser]      [Cotton, littlepolyster, weaving, handstitch, vcut]
4      [SilkShirt]          [wormsilk, artificialsilk, weaving, hand looming, color dying, coating]
5      [Carpet]             [Wool, cotton, organic cotton, knitting, sewing]

This time I have a lot of such data. There around 4200 such items in the Item column. I am trying to use TFIDF with bigrams and trigrams and using MultinomialNB to make the algorithm learn all the items in the Composition and predict the Item accurately.

1) I would like TF-IDF to so the n-grams at a word level so it can vectorise the words.
2) I would like the classifier to classify the Items based on the different compositions so it can learn what compositions put together.

For some items in Item column, they have around 10 values that comprise in the Composition column. So I used min_df=8 and ngram_range=(1,8), hoping it can try to build a vocabulary of around 8 words per composition.

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=8, norm='l2', encoding='latin-1', ngram_range=(1,8),     analyzer='word',lowercase=True,stop_words='english')
features = tfidf.fit_transform(dftest.Composition.values.astype('U')).toarray()
labels = dftest.Indexer

No matter what combinations of this I try the predictions are going wrong and I notice that the ngrams aren't really look at the word level in the Composition column.

I need expert advice on how to better solve this problem and mistakes in my approach. Consider me a novice in this area.


1 Answer 1


Not a very clever solution though. But I managed to do some trick to make it work. I am not fully satisfied by the result but the algorithm is able exactly predict the Item based on given Composition column.

from io import StringIO
import json
col = ['Item', 'Composition']
df = dfmin[col]
df['Item'] = df['Item'].apply(lambda x: ''.join(str(x).strip('[]') if      isinstance(x, list) else x))
df['Composition'] = df['Composition'].apply(lambda x: ''.join(str(x).strip('[]') if isinstance(x, list) else x))
df['Composition'] = df['Composition'].apply(lambda x: x.replace(',',' '))
df['Composition'] = df['Composition'].apply(lambda x: x.replace(' ',''))
df['Composition'] = df['Composition'].apply(lambda x: x.replace("'",''))
df['Composition'] = df['Composition'].apply(lambda x: "".join(x.rstrip()))

Doing this removed all spaces in the text in Composition column and it ended up becoming one long string set.

df['Indexer'] = df['Composition'].factorize()[0]
Indexer_dfmin = df[['Item',    'Indexer']].drop_duplicates().sort_values('Indexer')
df_to_Indexer = dict(Indexer_dfmin.values)
Indexer_to_df = dict(Indexer_dfmin[['Indexer', 'Item']].values)
#df['Composition'] = df['Composition'].str.split(",")

I factorized the Composition column using pd.factorize().

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(sublinear_tf=True, norm='l2', encoding='latin-1', ngram_range=(1,8), analyzer='word',lowercase=True)
features = tfidf.fit_transform(dftest.Composition.values.astype('U')).toarray()
labels = dftest.Indexer

Using sklearn RigdeClassifierCV I was able to model the data and predict the Item name using the string of the Composition column.

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import RidgeClassifierCV
X_train, X_test, y_train, y_test = train_test_split(df['Composition'].values.astype('U'),df['Item'].values.astype('U'), random_state=42)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf = RidgeClassifierCV(alphas=[1e-3, 1e-2, 1e-1, 1]).fit(X_train_tfidf, y_train)

Issues with this approach:
1) The model is not learning the words in the Composition column and building a vocabulary.
2) So, if I use the words with space from the Composition column the model will not be able to predict properly.
3) The accuracy can go near to 100% percent which means the model is overfit.

That is why I say this is not a clever solution. I would like to get to know how to fix the mistakes and issues in my model. Appreciate some expert advice on this. Please feel free to comment. Thanks in advance.

  • 1
    $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review $\endgroup$
    – Djib2011
    Oct 15, 2018 at 16:22
  • $\begingroup$ I wasn’t answering it. I didn’t receive help so I thought will post an update that I could fix it myself so that someone else doesn’t have to waste time fixing the answer for my question $\endgroup$
    – Sudhi
    Oct 15, 2018 at 16:33
  • $\begingroup$ you can post your own solution as an answer and accept it $\endgroup$
    – oW_
    Oct 15, 2018 at 19:46
  • $\begingroup$ @oW I have posted the answer now. I would appreciate some expert advice on the mistakes in my approach. $\endgroup$
    – Sudhi
    Oct 16, 2018 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.