2
$\begingroup$

I'm a high school senior who is new to data science, and would like to get into natural language processing. I currently know nothing about NLP, and the information online can be overwhelming. What are tokens? What are they used for / why do we need to tokenize text?

$\endgroup$
2
$\begingroup$

Well, I think it is better to watch a course about it. But, for instance, when people comment on different products on Amazon, there is a way to understand are their comments positive or negative, therefore their comments are used as data, analyzed and then labeled as positive or negative. Further, using this approach, one can find whether an email is spam or not, etc.

In order to analyze a text, we need to split it into its elements, meaning, its words (usually) each of these elements is called a token.

after splitting the text to words, useless words that do not represent anything, some signs, etc (the, ".", "my",...) are omitted, then we create bags of words, that are the most important words in the whole dataset (all comments), then each word is a column in the table and each row of the table is a comment and has 1 in a column if it contains that specific word. there is also a label column that shows if the comment is positive or not. then having this dataset, we can fit a classification model to the dataset.

$\endgroup$
1
$\begingroup$

"Tokens" are usually individual words (at least in languages like English) and "tokenization" is taking a text or set of text and breaking it up into its individual words

This is by far the simplest definition you can get about tokens. Consider a sentence as follows:- "Data is the new oil". Now humans can interpret the same sentence in a number of ways like - "We can call data as the new oil", "Data can also be called as the new oil". Now all these sentences almost mean the same and we as humans understand the sentence by reading the sentence word by word and associate the meaning it creates by putting it together.

As Deep learning is closely associated with replicating human-like methods, tokenization is one such method of simplifying the meaning of individual words and then creating a sense out of the sentence.

Read further about this at Tokenization tutorial

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.