4

The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using. Consider this example tweet: Hi, we need you! #Hi #Weneedyou If you use TreeBank or WordPunct tokenizers the output will be: ['Hi', 'we', 'need', 'you', '!', '#', 'Hi', '#', 'Weneedyou'] However if you use Whitespace Tokenizer, the result ...


3

It's possible if you define CountVectorizer's token_pattern argument. If you're new to regular expressions, Python's documentation goes over how it deals with regular expressions using the re module (and scikit-learn uses this under the hood) and I recommend using an online regex tester like this one, which gives you immediate feedback on whether your ...


3

Subword tokenization is the norm nowadays in NLP models because: It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. This is a problem for morphologically-rich languages, proper nouns, etc. Subword vocabularies allow representing these words. By having subword tokens (and ...


3

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used ...


3

Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence tagger and should behave better. Here is the issue with the details of its inception. You can train it if you have segmented sentence data. Another option is using NLTK's sent_tokenize, which should give better results than Spacy's ...


3

You can try with this: import pandas as pd import nltk df = pd.DataFrame({'frases': ['Do not let the day end without having grown a little,', 'without having been happy, without having increased your dreams', 'Do not let yourself be overcomed by discouragement.','We are passion-full beings.']}) df['tokenized'] = df.apply(lambda row: nltk.word_tokenize(row[...


3

Why would we want a custom tokenizer? Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer. Any toolkit needs to be flexible, and the ability to change the tokenizer, both so that someone can experiment, and so that it can be replaced if requirements are different, or better ways are found for specific problems, ...


2

For your problem i think gensim can be very useful, what can be implemented with Gensim library is phrase detection. It is similar to n-gram, but instead of getting all the n-gram by sliding the window, it detects frequently used phrases and stick them together. It statistically walks through the text corpus and identifies the common side-by-side occuring ...


2

The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation. One thing you can do is tokenize and tag all words with it's associated part-of-speech (PoS) tag, and then define regular expressions based on the PoS-tags to extract interesting key-...


2

You can change the tokenizer instead of the token pattern. The token pattern is used by the tokenizer. You can set as tokenizer any function that returns a list of string. For example, str.split tokenizer=str.split From the official documentation, tokenizer callable, default=None Override the string tokenization step while preserving the ...


2

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known). For a tfidf BOW model, you might get good performance from any of the above. For a model that is sensitive to word order I would certainly go with the first option and might ...


2

There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components To solve your problem, I see at least three ways to do it. NTLK NLTK allows you to add known abbreviations as exceptions. See this StackOverflow post. Use a regular expression Since your problem is that you have some example of dots that shouldn't ...


2

BPE and word pieces are fairly equivalent, with only minimal differences. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. Therefore, I understand that the authors of RoBERTa take the liberty of using BPE and wordpieces interchangeably.


2

The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked language model (MLM) and next sentence prediction (NSP). There were no other classification/tagging labels present in the data, as the MLM predicts the text itself ...


1

For your first question, you can check if the tokenizer covers a certain string with the following: text = 'today is a good day 😃' ids2string = lambda ids: tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids)) ids2string(tokenizer(text)['input_ids']) > <s>today is a good day 😃</s> If emoji is not included in the tokenizer ...


1

The explanation in the documentation of the Huggingface Transformers library seems more approachable: Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018). In contrast to BPE or WordPiece, Unigram initializes its base vocabulary to a large ...


1

There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here. You can see that it contains one token per line, with a total of 30522 tokens. The softmax is computed over them. The token granularity in the BERT vocabulary is subwords. ...


1

First, a clarification: normally, we distinguish tokenization and word segmentation. It is not clear to me what you exactly mean. Word segmentation is normally needed in languages without spaces between words, like Chinese. Tokenization is applied to most languages and is the process of splitting words in subword units to obtain a manageable vocabulary size ...


1

1 and 3 would be nice. Separating "well-known" to "well" and "known" would not be a good idea because you lost an information and/or have an erroneous/unuseful counts.


1

I agree with Nicholas' answer, a few more thoughts: you could use a standard English tokenizer (e.g. nltk, Spacy), if only to see how they process hyphenated words. Similarly you could check how it's done in a pre-tokenized dataset, but be aware that the tokenization conventions followed might differ from one dataset to the other. Imho the choice depends on ...


1

To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you. Examples include: NTLK Spacy SparkNLP .. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project. As for the detecting English ...


1

The problem is with the way things are documented. Check this link: https://stackoverflow.com/questions/46202519/keras-tokenizer-num-words-doesnt-seem-to-work


1

The tokenization process shouldn't be changed even when you are interested in multi words. After all, the words are still the basic tokens. What you should do it to find a way to combine the proper words into term. A simple way to do so is to look for term in which the probability of the term is higher than that of the independent tokens. For example P("...


Only top voted, non community-wiki answers of a minimum length are eligible