0
$\begingroup$

I have train/test data for my text classification problem. I have used them to create and test several ML models (LogisticRegression, RandomForest, and LinearSVC).

The train and test data consist of many documents classified into several categories. It is cleaned from dates and numbers, everything is lowercase and with no punctuation. Where the dates are cleaned I have substituted them with the word 'date'. The same approach I have applied to invoice numbers which were replaced with the word 'invoice'. This greatly helped my models because this specific word was given higher weight and it improved classification.

Now that I have chosen the best model I plan to use it for the new data that will be coming. As for this new data, should I clean it before it goes to the trained model (as I clean my train/test data), or am I supposed to leave it as is?

$\endgroup$
1
  • 1
    $\begingroup$ You should feed the trained model data in the same format as the data that it was trained on, which means that you need to apply the same preprocessing steps before making a prediction with the trained model. $\endgroup$
    – Oxbowerce
    Sep 1, 2022 at 14:31

1 Answer 1

1
$\begingroup$

Yes, it makes perfect sense to clean/preprocess the new data much like train /test dataset.

For reference:

[https://stackoverflow.com/questions/66301306/do-you-have-to-clean-your-test-data-before-feeding-into-an-nlp-model][1]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.