Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

What are the inputs of encoder and decoder layers of transformer architecture?

In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
user avatar
  • 45
0 votes
0 answers
12 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
user avatar
  • 259
1 vote
1 answer
26 views

How to deal with "Ergänzungsstrichen" and "Bindestrichen" in German NLP?

Problem In German, the phrase "Haupt- und Nebensatz" has exactly the same meaning as "Hauptsatz und Nebensatz". However, when transforming both phrases using e.g. spacy's ...
user avatar
1 vote
1 answer
29 views

does ValueError: 'rat' is not in list means not exist in tokenizer

Does this error means that the word doesn't exist in the tokenizer return sent.split(" ").index(word) ValueError: 'rat' is not in list the code sequences ...
user avatar
0 votes
0 answers
13 views

Any way to make NER tagging with float(2.0) and inferencing with str(2)

One of the NER attribute is tagged with float (3.0, 2.0, ...) while the text file I am trying to inference from are in string format of (3, 2, ...). The Spacy model I used can't pick up the numbers ...
user avatar
0 votes
2 answers
97 views

How does Keras Tokenizer choose tokens given a sentence?

I tried to find the answer to this question but I can't find anything, so I ask here: How does Keras Tokenizer choose tokens given a sentence of words ? To be more precise with what I want to know, ...
user avatar
1 vote
1 answer
39 views

How to precompute one sequence in a sequence-pair task when using BERT?

BERT uses separator tokens ([SEP]) to input two sequences for a sequence-pair task. If I understand the BERT architecture correctly, attention is applied to all inputs thus coupling the two sequences ...
user avatar
2 votes
1 answer
647 views

What is the difference between TextVectorization and Tokenizer?

What is the difference between the layers.TextVectorization() and ...
user avatar
0 votes
1 answer
499 views

Adding a new token to a transformer model without breaking tokenization of subwords

I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
user avatar
  • 101
1 vote
0 answers
19 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
user avatar
0 votes
0 answers
22 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
user avatar
0 votes
1 answer
13 views

Training NMT models for noisy social media roman text

I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the ...
user avatar
  • 131
0 votes
0 answers
10 views

Tensorflow text tokenizer incorrect tokenization

I am trying to use TF Tokenizer for a NLP model ...
user avatar
1 vote
1 answer
22 views

How is the connection between Text Mining, NLP and Tasks like Tokenization, Lemmatization, Stop-word Removal etc.?

I am new to the whole world around Big Data and Text Mining. It took me a while to understand all the connections and to be able to classify the buzzwords. But there's one thing I still don't ...
user avatar
  • 27
0 votes
0 answers
13 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
user avatar
  • 285
0 votes
0 answers
26 views

An efficient way to resegment tokenized text into phrases

I have text tokenized on the word level and few lists of phrases stored as tuples. What would be the most efficient way to resegment (and store) the text into phrases? For example, a sentence like: &...
user avatar
  • 1
0 votes
0 answers
52 views

Character Level Embedding in Sentence Classification

I'm working on an NLP task that requires the use of character level embeddings. By using tokenizer library I realized that it tokenizes such as lower integer meant the most frequent character. Is ...
user avatar
0 votes
1 answer
340 views

What tokenizer does OpenAI's GPT3 API use?

I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-...
user avatar
0 votes
1 answer
274 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
user avatar
  • 31
1 vote
1 answer
166 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
user avatar
1 vote
1 answer
16 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
user avatar
4 votes
1 answer
2k views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
user avatar
  • 165
2 votes
1 answer
578 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
user avatar
  • 121
5 votes
2 answers
1k views

Converting paragraphs into sentences

I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's ...
user avatar
  • 285
0 votes
1 answer
1k views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
user avatar
  • 117
2 votes
1 answer
191 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
user avatar
1 vote
1 answer
302 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
user avatar
  • 117
1 vote
1 answer
113 views

Why BERT tokenizers function differently?

While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer: ...
user avatar
-1 votes
1 answer
167 views

How does a neural tokenizer work? [closed]

I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens. But it is not clear to me how this kind of model should work in terms of the output format. If the ...
user avatar
3 votes
1 answer
985 views

NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?

I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text ...
user avatar
  • 309
0 votes
2 answers
649 views

Is it good practice to remove the numeric values from the text data during preprocessing?

Im doing preprocessing on a text dataset. I have certain numerics in it like: date(1st July) year(2019) tentative values (3-5 years/ 10+ advantages). unique values (room no 31/ user rank 45) ...
user avatar
  • 23
1 vote
3 answers
898 views

How to process the hyphenated english words for any nlp problem?

Im doing preprocessing on english text dataset. I encounter hyphenated words like 'well-known'. Will it be useful if I remove the hyphen as special character and treat it as a single word 'wellknown' ...
user avatar
  • 23
1 vote
1 answer
30 views

create sequence of non dictionary words

I have a few word vectors- recvfrom,sendto,epoll_pwait,recvfrom,sendto,epoll_pwait getuid,recvfrom,writev,getuid,epoll_pwait,getuid Now i want to tokenized them ...
user avatar
1 vote
1 answer
192 views

Why does my char level Keras tokenizer add spaces when converting sequences to texts?

I create a tokenizer with import tf tokenizer = tf.keras.preprocessing.text.Tokenizer(split='', char_level=True, ...) tokenizer.fit_to_texts(...) But when I ...
user avatar
1 vote
0 answers
70 views

CountVectorizer vs HashVectorizer for text

I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work ...
user avatar
  • 49
0 votes
3 answers
6k views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
user avatar
0 votes
1 answer
119 views

How to Calculate semantic similarity between video captions?

I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences. For example, the captions for one video are as follows: These captions are for the same ...
user avatar
2 votes
1 answer
13k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
user avatar
  • 45
1 vote
4 answers
4k views

how to avoid tokenizing w/ sklearn feature extraction

I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be ...
user avatar
  • 23
1 vote
2 answers
483 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
user avatar
1 vote
1 answer
914 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
user avatar
  • 13
2 votes
1 answer
784 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
user avatar
  • 293
1 vote
0 answers
216 views

simplifying AND OR Boolean Expression [closed]

My problem is turning a string that looks like this. "a OR (b AND c)" into a OR bc if the expression is like ...
user avatar
  • 201
1 vote
0 answers
181 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
user avatar
7 votes
1 answer
9k views

Understanding the effect of num_words of Tokenizer in Keras

Consider the following code: ...
user avatar
  • 257
2 votes
1 answer
2k views

How to customize word division in CountVectorizer?

...
user avatar
0 votes
1 answer
1k views

How do NLP tokenizers handle hashtags?

I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?
user avatar
5 votes
1 answer
1k views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
user avatar
  • 111
9 votes
6 answers
12k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...
user avatar