Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
30 views

How is tokenization done in pretrained word2vec models supplied by Google

I came across the pre-trained word2vec supplied by google at https://code.google.com/archive/p/word2vec/ (https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing) this gives a ...
0
votes
3answers
476 views

how to avoid tokenizing w/ sklearn feature extraction

I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be ...
0
votes
2answers
124 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
1answer
326 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
1
vote
1answer
148 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
1
vote
0answers
131 views

simplifying AND OR Boolean Expression [closed]

My problem is turning a string that looks like this. "a OR (b AND c)" into a OR bc if the expression is like ...
1
vote
0answers
152 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
2
votes
1answer
2k views

Understanding the effect of num_words of Tokenizer in Keras

Consider the following code: ...
0
votes
1answer
521 views

How do NLP tokenizers handle hashtags?

I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?
4
votes
1answer
781 views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
7
votes
6answers
7k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...