Here's a high-level blueprint of my model -

Input data::

CATEGORY    SERVICE     TITLE                               DEPARTMENT

apple       fruits      i love eating fruits.               fruitshop
mango       fruits      mangoes are yellow in color.        fruitshop
cycle       vehicle     that cycle is really expensive.     motorshop   

I convert this to a sparse matrix with Tf-Idf scores, where the columns are the title columns tokenized.

I am using LinearSVC and StratifiedKFold (n_splits = 10, random_state=777, shuffle=True) able to achieve a prediction accuracy of 73%.

Got 2 doubts -

1) What happens when a string like "fresh fish" is used for classification? Because the words "fresh" and "fish" are never used before, which department will this be classified into?

2) How can I consume this string "fresh fish" in my model?

Currently, I have something like this -

# X.shape = (1181, 1930)
# y.shape = (1181,)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=777)
pr = mod.predict(X_test)
print(pd.DataFrame(confusion_matrix(y_test, pr)))

I tried giving input as print(mod.predict([["fresh fish"]])) but got an error -

ValueError: X has 1 features per sample; expecting 1930

Please advise. Thanks.


You have to repeat the same preprocessing steps for prediction as you did during training. Let's say you trained like this:

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=777)

vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(X_train)

mod = LinearSVC()
mod.fit(X_train, y_train)

Then you need to use that fit TfidfVectorizer to transform your test prediction as well (which you seem to have managed). Like this:

X_test = vectorizer.transform(X_test)
pr = mod.predict(X_test)

And the same goes if you just want to feed in a manual text:

X_manual = vectorizer.transform(['fresh fish'])
pr = mod.predict(X_manual)

Sklearn pipeline can also be used, this will avoid you writing preprocessing step again.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC

a= [(' i love eating fruits','fruitshop'),
    ('mangoes are yellow in color','fruitshop'),
    ('that cycle is really expensive','motorshop')]

df = pd.DataFrame(a, columns = ['title', 'Department'])

model = Pipeline([('tfidf', TfidfVectorizer()), 


model.predict(['fresh fish'])
array(['fruitshop'], dtype=object)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.