Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
67 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
1
vote
1answer
44 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
1
vote
1answer
15 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
1
vote
1answer
271 views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
1
vote
1answer
163 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
4
votes
2answers
324 views

Converting paragraphs into sentences

I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's ...
0
votes
1answer
381 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
1
vote
1answer
87 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
1
vote
1answer
171 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
1answer
55 views

Why BERT tokenizers function differently?

While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer: ...
-1
votes
1answer
37 views

How does a neural tokenizer work? [closed]

I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens. But it is not clear to me how this kind of model should work in terms of the output format. If the ...
1
vote
1answer
515 views

NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?

I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text ...
0
votes
0answers
38 views

Tensorflow tokeniser: the maximum number of words to keep

Trying to tokenize the IMDB movie reviews by applying Tensorflow tokenizer. I want to have a maximum 10000-word vocabulary. For unseen words, I use a default token. ...
0
votes
1answer
55 views

Is it good practice to remove the numeric values from the text data during preprocessing?

Im doing preprocessing on a text dataset. I have certain numerics in it like: date(1st July) year(2019) tentative values (3-5 years/ 10+ advantages). unique values (room no 31/ user rank 45) ...
1
vote
3answers
291 views

How to process the hyphenated english words for any nlp problem?

Im doing preprocessing on english text dataset. I encounter hyphenated words like 'well-known'. Will it be useful if I remove the hyphen as special character and treat it as a single word 'wellknown' ...
1
vote
1answer
29 views

create sequence of non dictionary words

I have a few word vectors- recvfrom,sendto,epoll_pwait,recvfrom,sendto,epoll_pwait getuid,recvfrom,writev,getuid,epoll_pwait,getuid Now i want to tokenized them ...
1
vote
0answers
100 views

Why does my char level Keras tokenizer add spaces when converting sequences to texts?

I create a tokenizer with import tf tokenizer = tf.keras.preprocessing.text.Tokenizer(split='', char_level=True, ...) tokenizer.fit_to_texts(...) But when I ...
0
votes
0answers
19 views

How does the ULM subword tokenization avoid just splitting every word into single characters?

According to the paper: The probability of a subword sequence X = (x_1, x_2...) is formulated as the product of the subword occurence probabilities p(x_i) sum_i p(x_i) = 1 The most probable ...
1
vote
0answers
38 views

CountVectorizer vs HashVectorizer for text

I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work ...
0
votes
3answers
3k views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
0
votes
1answer
59 views

How to Calculate semantic similarity between video captions?

I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences. For example, the captions for one video are as follows: These captions are for the same ...
2
votes
1answer
8k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
0
votes
4answers
3k views

how to avoid tokenizing w/ sklearn feature extraction

I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be ...
1
vote
2answers
388 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
1answer
699 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
1
vote
1answer
592 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
1
vote
0answers
206 views

simplifying AND OR Boolean Expression [closed]

My problem is turning a string that looks like this. "a OR (b AND c)" into a OR bc if the expression is like ...
1
vote
0answers
165 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
7
votes
1answer
6k views

Understanding the effect of num_words of Tokenizer in Keras

Consider the following code: ...
2
votes
1answer
2k views
0
votes
1answer
1k views

How do NLP tokenizers handle hashtags?

I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?
5
votes
1answer
1k views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
8
votes
6answers
11k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...