1
$\begingroup$

I have a problem. I have a NLP classification problem. There are different methods to decompose sentences into tokens, for example in whole words or in characters. Then there are different tokenizers like:

  • TF-IDF
  • Binary
  • Frequency
  • Count

My question now aims, why should one make the effort and use a different word division (word or character) and then check this with the different tokenizers?

$\endgroup$

2 Answers 2

2
$\begingroup$
  • It's rare to represent sentences as sequences of characters, since most NLP tasks are related to the the semantics of the sentence, which is expressed by the sequence of words. A notable exception: stylometry tasks, i.e. tasks where the style of the text/author matters more than the topic/meaning, sometimes rely on sequences of characters.
  • Yes, the question of tokenization can indeed have an impact of the performance of the target task. But modern methods use good word tokenizers trained on large corpora, not simplifed whitespace-based tokenizers. There can still be differences between tokenizers though.
  • There are even more text representations methods than listed here (embeddings are an important one). And yes, these also have a huge impact on performance.

For all these different options (and others), the reason why it's often worth testing different variants is clear: it affects performance and it's not always clear which one is the best without trying, so one must evaluate the different options. Btw it's crucial to precisely define how the target task is evaluated first, otherwise one just subjectively interprets results.

Basically imho this is a matter of proper data-driven methodology. Of course experience and intuition also play a role, especially if there are time or resources constrains.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer! Would you also have a scientific source / paper on this? So that one can read more or the statement proves? $\endgroup$
    – Test
    Oct 23, 2022 at 13:49
  • 1
    $\begingroup$ @Test On the specific topic of word tokenization, I happen to have authored this paper a few years ago which I think gives a decent overview. On the broad topic of selecting the best method based on a well-defined evaluation methodology, I'm not sure what to recommend in particular: virtually every NLP paper deals with this kind of question, to some extent. $\endgroup$
    – Erwan
    Oct 23, 2022 at 16:36
1
$\begingroup$

The reason to explore different tokenization and vectorization methods is that those modeling choices might impact classification performance. The best practice is to train with different values for those hyperparameters and see which combination has the highest evaluation metric performance on a hold-out dataset.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer! Would you also have a scientific source / paper on this? So that one can read more or the statement proves? $\endgroup$
    – Test
    Oct 23, 2022 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.