0
$\begingroup$

So basically my question is hypothetically lets say:

I have a column containing 2000 rows of texts, and when I apply tf-idf, I get 27 features like shown below.

enter image description here

Now once I do that, I could consider my Neural Network's Input layer's number of neurons to be 27, like shown below, and i train the model with the tf-idf features.

enter image description here

Now, hypothetically speaking, if I'm trying to test this model with one string (a short string), and when we apply Tf-Idf to that string text, we get 20 features. Now this number of features does not equal to the no of neurons in my input layer, which is 27, which can cause problems.

How can we tackle a problem like this, I have seen we can use max no of features with Tf-Idf, but i thought it would be good to ask the community as well, so that you could show me a better way.

Is there a way we could have a fixed length of features when applying Tf-Idf so that there wont be a problem when feeding to the neural network.

Your help will be appreciated!

$\endgroup$

1 Answer 1

0
$\begingroup$

If you want to apply the same step on new data you would call the transform method of the tf-idf vectorizer, which should give you the same number of features. See the example below:

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
text = [
    "This is a test string.",
    "This is another test string."
]
print("Shape of the transformed training sentences:", vectorizer.fit_transform(text).shape)
print("Shape of the new string to test the model on:", vectorizer.transform(["A string to test the model on."]).shape)

# Shape of the transformed training sentences: (2, 5)
# Shape of the new string to test the model on: (1, 5)

As you can see, the number of features (columns) stays the same (5) when applying the trained vectorizer on new data.

$\endgroup$
5
  • $\begingroup$ Wait, so can we export a vectorizer or rather save a vectorizer just like we can save a model (h5 file) as a file? Because the above way would be impossible if we export my NN model and host it, how can I call the same vectorizer? @Oxbowerce $\endgroup$ Feb 20, 2022 at 16:13
  • $\begingroup$ Correct, you can simply save the vectorizer (or any other preprocessing steps by using a pipeline) into a file which you can reload later and apply on new data once you would start using your model. You could save the vectorizer using pickle or joblib, see also the scikit-learn documentation on model persistence. $\endgroup$
    – Oxbowerce
    Feb 20, 2022 at 16:22
  • $\begingroup$ Ok so once i save it, no matter what sentence i use for testing, it will give the same no of features (columns) which it got when we add the training data at the very first time? $\endgroup$ Feb 20, 2022 at 16:37
  • $\begingroup$ That is indeed correct, however you need to make sure that you use the transform method instead of fit_transform as the latter would refit the vectorizer on the new data. This is not what you want, as you want to use the learned words from the training dataset and transform any new data based on that information. $\endgroup$
    – Oxbowerce
    Feb 20, 2022 at 19:29
  • $\begingroup$ Got it, thnx a lot! $\endgroup$ Feb 21, 2022 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.