4

The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using. Consider this example tweet: Hi, we need you! #Hi #Weneedyou If you use TreeBank or WordPunct tokenizers the output will be: ['Hi', 'we', 'need', 'you', '!', '#', 'Hi', '#', 'Weneedyou'] However if you use Whitespace Tokenizer, the result ...


3

It's possible if you define CountVectorizer's token_pattern argument. If you're new to regular expressions, Python's documentation goes over how it deals with regular expressions using the re module (and scikit-learn uses this under the hood) and I recommend using an online regex tester like this one, which gives you immediate feedback on whether your ...


2

The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation. One thing you can do is tokenize and tag all words with it's associated part-of-speech (PoS) tag, and then define regular expressions based on the PoS-tags to extract interesting key-...


2

Why would we want a custom tokenizer? Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer. Any toolkit needs to be flexible, and the ability to change the tokenizer, both so that someone can experiment, and so that it can be replaced if requirements are different, or better ways are found for specific problems, ...


2

Subword tokenization is the norm nowadays in NLP models because: It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. This is a problem for morphologically-rich languages, proper nouns, etc. Subword vocabularies allow representing these words. By having subword tokens (and ...


2

You can try with this: import pandas as pd import nltk df = pd.DataFrame({'frases': ['Do not let the day end without having grown a little,', 'without having been happy, without having increased your dreams', 'Do not let yourself be overcomed by discouragement.','We are passion-full beings.']}) df['tokenized'] = df.apply(lambda row: nltk.word_tokenize(row[...


1

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used ...


1

First, a clarification: normally, we distinguish tokenization and word segmentation. It is not clear to me what you exactly mean. Word segmentation is normally needed in languages without spaces between words, like Chinese. Tokenization is applied to most languages and is the process of splitting words in subword units to obtain a manageable vocabulary size ...


1

The problem is with the way things are documented. Check this link: https://stackoverflow.com/questions/46202519/keras-tokenizer-num-words-doesnt-seem-to-work


1

1 and 3 would be nice. Separating "well-known" to "well" and "known" would not be a good idea because you lost an information and/or have an erroneous/unuseful counts.


1

I agree with Nicholas' answer, a few more thoughts: you could use a standard English tokenizer (e.g. nltk, Spacy), if only to see how they process hyphenated words. Similarly you could check how it's done in a pre-tokenized dataset, but be aware that the tokenization conventions followed might differ from one dataset to the other. Imho the choice depends on ...


1

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known). For a tfidf BOW model, you might get good performance from any of the above. For a model that is sensitive to word order I would certainly go with the first option and might ...


1

To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you. Examples include: NTLK Spacy SparkNLP .. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project. As for the detecting English ...


1

For your problem i think gensim can be very useful, what can be implemented with Gensim library is phrase detection. It is similar to n-gram, but instead of getting all the n-gram by sliding the window, it detects frequently used phrases and stick them together. It statistically walks through the text corpus and identifies the common side-by-side occuring ...


1

The tokenization process shouldn't be changed even when you are interested in multi words. After all, the words are still the basic tokens. What you should do it to find a way to combine the proper words into term. A simple way to do so is to look for term in which the probability of the term is higher than that of the independent tokens. For example P("...


Only top voted, non community-wiki answers of a minimum length are eligible