1
$\begingroup$

This is a multiclass text classification problem. The dataset has a class imbalance and I'm planning to use a sampling technique before modeling.

Should the sampling be done before/after the TFIDVectorizer step? Kindly share your thoughts.

from imblearn.pipeline import Pipeline
from imblearn.over_sampling import RandomOverSampler, SMOTE
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = ...

############# LIKE THIS #############
ppl = Pipeline(steps=[
    ('vectorizer', TfidfVectorizer(ngram_range=(1, 2))),
    ('sampler', SMOTE()),
    ('classifier', LogisticRegression(max_iter=1000, n_jobs=-1),
)]
ppl.fit(X_train, y_train)

############# OR LIKE THIS #############
sampler = SMOTE() # SMOTE couldn't be put inside pipeline before TFIDF step because its output format is incompatible with TFidf, hence moved outside of pipeline.
X_train, y_train = sm.fit_sample(X_train, y_train)

ppl = Pipeline(steps=[
    ('vectorizer', TfidfVectorizer(ngram_range=(1, 2))),
    ('classifier', LogisticRegression(max_iter=1000, n_jobs=-1),
)]
ppl.fit(X_train, y_train)

$\endgroup$

1 Answer 1

1
$\begingroup$

First, let me note that resampling rarely works well with text data, because the diversity of text cannot be simulated by some extremely simplistic process like SMOTE. Of course you can try resampling, but don't forget to compare to the basic case without resampling.

Now, the difference between your two options would be the IDF weights in TFIDF, because the IDF for a token depends on the number of instances which contain this token.

  • Option 1 with TFIDF before SMOTE: using the true IDF weights corresponding to the original text. Disadvantage: not consistent with the modified frequencies caused by SMOTE.
  • Option 2 with SMOTE before TFIDF: using IDF weights after changing frequencies, so consistent with these. Disadvantage: IDF weights are artificial and more likely to introduce a bias. For example a rare word like "overfitting" might be duplicated many time, hence its frequency is higher and its IDF is lower.

I think that option 1 is preferable, the representation of the text is more faithful to the real data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.