Why is it useful to use different word splitting with different tokenizers?

Question

I have a problem. I have a NLP classification problem. There are different methods to decompose sentences into tokens, for example in whole words or in characters. Then there are different tokenizers like:

• TF-IDF
• Binary
• Frequency
• Count

My question now aims, why should one make the effort and use a different word division (word or character) and then check this with the different tokenizers?

Answer 1

• It's rare to represent sentences as sequences of characters, since most NLP tasks are related to the the semantics of the sentence, which is expressed by the sequence of words. A notable exception: stylometry tasks, i.e. tasks where the style of the text/author matters more than the topic/meaning, sometimes rely on sequences of characters.
• Yes, the question of tokenization can indeed have an impact of the performance of the target task. But modern methods use good word tokenizers trained on large corpora, not simplifed whitespace-based tokenizers. There can still be differences between tokenizers though.
• There are even more text representations methods than listed here (embeddings are an important one). And yes, these also have a huge impact on performance.

For all these different options (and others), the reason why it's often worth testing different variants is clear: it affects performance and it's not always clear which one is the best without trying, so one must evaluate the different options. Btw it's crucial to precisely define how the target task is evaluated first, otherwise one just subjectively interprets results.

Basically imho this is a matter of proper data-driven methodology. Of course experience and intuition also play a role, especially if there are time or resources constrains.

Answer 2

The reason to explore different tokenization and vectorization methods is that those modeling choices might impact classification performance. The best practice is to train with different values for those hyperparameters and see which combination has the highest evaluation metric performance on a hold-out dataset.