Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
17 views

does ValueError: 'rat' is not in list means not exist in tokenizer

Does this error means that the word doesn't exist in the tokenizer return sent.split(" ").index(word) ValueError: 'rat' is not in list the code sequences ...
0
votes
0answers
12 views

Any way to make NER tagging with float(2.0) and inferencing with str(2)

One of the NER attribute is tagged with float (3.0, 2.0, ...) while the text file I am trying to inference from are in string format of (3, 2, ...). The Spacy model I used can't pick up the numbers ...
0
votes
2answers
19 views

How does Keras Tokenizer choose tokens given a sentence?

I tried to find the answer to this question but I can't find anything, so I ask here: How does Keras Tokenizer choose tokens given a sentence of words ? To be more precise with what I want to know, ...
1
vote
1answer
22 views

How to precompute one sequence in a sequence-pair task when using BERT?

BERT uses separator tokens ([SEP]) to input two sequences for a sequence-pair task. If I understand the BERT architecture correctly, attention is applied to all inputs thus coupling the two sequences ...
2
votes
1answer
94 views

What is the difference between TextVectorization and Tokenizer?

What is the difference between the layers.TextVectorization() and ...
0
votes
1answer
27 views

Adding a new token to a transformer model without breaking tokenization of subwords

I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
1
vote
0answers
16 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
0
votes
0answers
14 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
0
votes
1answer
12 views

Training NMT models for noisy social media roman text

I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the ...
0
votes
0answers
9 views

Tensorflow text tokenizer incorrect tokenization

I am trying to use TF Tokenizer for a NLP model ...
1
vote
1answer
19 views

How is the connection between Text Mining, NLP and Tasks like Tokenization, Lemmatization, Stop-word Removal etc.?

I am new to the whole world around Big Data and Text Mining. It took me a while to understand all the connections and to be able to classify the buzzwords. But there's one thing I still don't ...
0
votes
0answers
13 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
0
votes
0answers
23 views

An efficient way to resegment tokenized text into phrases

I have text tokenized on the word level and few lists of phrases stored as tuples. What would be the most efficient way to resegment (and store) the text into phrases? For example, a sentence like: &...
0
votes
0answers
52 views

Character Level Embedding in Sentence Classification

I'm working on an NLP task that requires the use of character level embeddings. By using tokenizer library I realized that it tokenizes such as lower integer meant the most frequent character. Is ...
0
votes
0answers
63 views

What tokenizer does OpenAI's GPT3 API use?

I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-...
0
votes
1answer
224 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
1
vote
1answer
107 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
1
vote
1answer
16 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
3
votes
1answer
947 views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
2
votes
1answer
372 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
5
votes
2answers
691 views

Converting paragraphs into sentences

I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's ...
0
votes
1answer
812 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
1
vote
1answer
154 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
1
vote
1answer
257 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
1answer
81 views

Why BERT tokenizers function differently?

While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer: ...
-1
votes
1answer
71 views

How does a neural tokenizer work? [closed]

I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens. But it is not clear to me how this kind of model should work in terms of the output format. If the ...
2
votes
1answer
802 views

NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?

I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text ...
0
votes
1answer
258 views

Is it good practice to remove the numeric values from the text data during preprocessing?

Im doing preprocessing on a text dataset. I have certain numerics in it like: date(1st July) year(2019) tentative values (3-5 years/ 10+ advantages). unique values (room no 31/ user rank 45) ...
1
vote
3answers
632 views

How to process the hyphenated english words for any nlp problem?

Im doing preprocessing on english text dataset. I encounter hyphenated words like 'well-known'. Will it be useful if I remove the hyphen as special character and treat it as a single word 'wellknown' ...
1
vote
1answer
30 views

create sequence of non dictionary words

I have a few word vectors- recvfrom,sendto,epoll_pwait,recvfrom,sendto,epoll_pwait getuid,recvfrom,writev,getuid,epoll_pwait,getuid Now i want to tokenized them ...
1
vote
0answers
135 views

Why does my char level Keras tokenizer add spaces when converting sequences to texts?

I create a tokenizer with import tf tokenizer = tf.keras.preprocessing.text.Tokenizer(split='', char_level=True, ...) tokenizer.fit_to_texts(...) But when I ...
1
vote
0answers
49 views

CountVectorizer vs HashVectorizer for text

I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work ...
0
votes
3answers
5k views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
0
votes
1answer
98 views

How to Calculate semantic similarity between video captions?

I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences. For example, the captions for one video are as follows: These captions are for the same ...
2
votes
1answer
11k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
1
vote
4answers
3k views

how to avoid tokenizing w/ sklearn feature extraction

I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be ...
1
vote
2answers
431 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
1answer
814 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
1
vote
1answer
718 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
1
vote
0answers
209 views

simplifying AND OR Boolean Expression [closed]

My problem is turning a string that looks like this. "a OR (b AND c)" into a OR bc if the expression is like ...
1
vote
0answers
171 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
7
votes
1answer
8k views

Understanding the effect of num_words of Tokenizer in Keras

Consider the following code: ...
2
votes
1answer
2k views
0
votes
1answer
1k views

How do NLP tokenizers handle hashtags?

I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?
5
votes
1answer
1k views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
9
votes
6answers
11k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...