Questions tagged [tokenization]
The tokenization tag has no usage guidance.
69
questions
0
votes
0
answers
6
views
How to pre-processed text to keep dashed words?
I'm working on compiling reviews for movies and analysing it in Orange. I've found that words like "r-rated" get converted to "r" and "rated". I've messed around a bit ...
- 1
0
votes
0
answers
24
views
Sentencepiece Tokenizer training from scrath
To train BPE model on sentencepiece as per given Usage instructions
As, it is mentioned in the instructions that --input: one-sentence-per-line raw corpus file.
...
- 167
0
votes
1
answer
31
views
Why is the sprase categorical accuracy decreasing every epoch and predictions are always NaN?
Problem Summary
My model is built and compiled properly but gets the NaN validation loss on all epochs. The training set accuracy is also infinitesimally small and keeps decreasing. I couldn't find a ...
- 113
0
votes
0
answers
27
views
How to improve GPT2 tokenizer trained from scratch?
I trained a GPT2 Tokenizer on Hindi dataset of size 170 MB from scratch and saved it as new_tokenizer. When I tried the new_tokenizer on a Hindi sentence
...
- 167
0
votes
0
answers
39
views
Help understanding working of KeyBERT for keyphrase extraction
I'm fairly new to reading and understanding research papers, so I wanted to get a second opinion on whether my understanding of KeyBERT was correct. Here is a high level overview of my understanding ...
- 33
1
vote
2
answers
466
views
how do we adapt LLM token embeddings with custom vocab
Hi im just getting started with understanding transformer based models and I am not able to find how the token embeddings are arrived at?. there are multiple tokenization approaches and multiple ...
- 111
2
votes
2
answers
268
views
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need&...
2
votes
1
answer
67
views
How was the token library constructed for ChatGPT / other GPT systems?
I have found literally hundreds of articles on Google with titles like 'What are tokens and how to use them,' but haven't been able to find any information at all on how the token libraries themselves ...
- 153
1
vote
1
answer
63
views
What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?
I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
4
votes
2
answers
16k
views
ChatGPT: How to use long texts in prompt?
I like the website chatpdf.com a lot. You can upload a PDF file and then discuss the textual content of the file with the file "itself". It uses ChatGPT.
I would like to program something ...
- 63
0
votes
0
answers
27
views
How do you tokenize and/or parse variable labels in a language which allows variable labels to have a digit on the left-side?
How do you tokenize and/or parse variable names in a language which allows variable names to have a digit 0, 1, ...
- 101
0
votes
1
answer
166
views
What does Codex take as tokens?
The typical default for neural networks in natural language processing has been to take words as tokens.
OpenAI Codex is based on GPT-3, but also deals with source code. For source code in general, ...
- 139
0
votes
0
answers
25
views
Does order matter in this causal language model?
Say you've implemented a causal language model like so:
...
- 1
2
votes
1
answer
202
views
Does GPT-3 remember data from prompts used to fine tune it?
I am trying to fine tune a model using OpenAI's fine tuning API. I am passing bodies of text (for example, news paper articles) as prompts and the data I want from it as completions.
Let us consider ...
0
votes
0
answers
39
views
How to tokenize custom dataset?
I have generated a dataset of quadratic equations and I want to train T5 model to solve them.
Dataset looks like this:
Could somebody please give a hint on how to tokenize this correctly? What to do ...
1
vote
2
answers
39
views
Why is it useful to use different word splitting with different tokenizers?
I have a problem. I have a NLP classification problem.
There are different methods to decompose sentences into tokens, for example in whole words or in characters. Then there are different tokenizers ...
- 89
0
votes
1
answer
100
views
Advantages of different tokenizers for NLP (specifically text generation)
What are the advantages of using different tokenizers? For example, let's take the sentence:
"In Düsseldorf I took my hat off. But I can't put it back on."
The treebank tokenizer yields: &...
0
votes
1
answer
65
views
What counts as a token for bpemb's encode_ids_with_eos()
I have probelms understanding bpemb's encode_ids_with_eos() or similar.
When I run the following code i get none-word like segmentations (rather syllalbus based or ...
- 125
0
votes
1
answer
108
views
Why GloVe model (by gensim) does not have vectors for numbers 1, 2, ...?
I expected GLoVe to have vectors for numbers.
from gensim import downloader as api
glove = api.load("glove-twitter-25")
glove['1']
This results in ...
- 101
1
vote
1
answer
50
views
0
votes
0
answers
418
views
Slow and Fast tokenizer gives different outputs(sentencepiece tokenizer)
When i use T5TokenizerFast(Tokenizer of T5 arcitecture), the output is expected as follows:
['▁', '</s>', '▁Hello', '▁', '<sep>', '</s>']
But ...
- 91
0
votes
1
answer
158
views
Smaller embedding size causes lower loss
When I convert my multilingual transformer model to a single lingual transformer model (got my languages embedding from the multilingual transformer and deleted other embeddings, decreased dimensions ...
- 91
0
votes
1
answer
52
views
What is the difference between adding words to a tokenizer and training a tokenizer?
The title says it all. I was researching this question but couldn't find something useful. What is the difference between adding words to a tokenizer and training a tokenizer?
- 91
0
votes
1
answer
35
views
What is the effect of the tokens?
What is the effect of the tokens that the model has if model A has 1B tokens and the other model has 12B tokens? Will that have an effect on the performance?
- 57
1
vote
0
answers
408
views
What are the inputs of encoder and decoder layers of transformer architecture?
In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
- 91
1
vote
1
answer
104
views
How to deal with "Ergänzungsstrichen" and "Bindestrichen" in German NLP?
Problem
In German, the phrase "Haupt- und Nebensatz" has exactly the same meaning as "Hauptsatz und Nebensatz". However, when transforming both phrases using e.g. spacy's ...
- 193
1
vote
1
answer
101
views
does ValueError: 'rat' is not in list means not exist in tokenizer
Does this error means that the word doesn't exist in the tokenizer
return sent.split(" ").index(word)
ValueError: 'rat' is not in list
the code sequences ...
- 17
0
votes
2
answers
493
views
How does Keras Tokenizer choose tokens given a sentence?
I tried to find the answer to this question but I can't find anything, so I ask here: How does Keras Tokenizer choose tokens given a sentence of words ?
To be more precise with what I want to know, ...
1
vote
1
answer
131
views
How to precompute one sequence in a sequence-pair task when using BERT?
BERT uses separator tokens ([SEP]) to input two sequences for a sequence-pair task. If I understand the BERT architecture correctly, attention is applied to all inputs thus coupling the two sequences ...
3
votes
1
answer
2k
views
What is the difference between TextVectorization and Tokenizer?
What is the difference between the layers.TextVectorization() and
...
- 193
1
vote
2
answers
3k
views
Adding a new token to a transformer model without breaking tokenization of subwords
I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
- 111
1
vote
0
answers
24
views
Tokenizer returning incorrect values and losing a lot of data
(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
0
votes
0
answers
51
views
When to do tokenization and does my output need tokenization after stemming?
I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews.
So first thing i did is removing special characters, white spaces, ...
0
votes
1
answer
16
views
Training NMT models for noisy social media roman text
I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the ...
- 131
1
vote
1
answer
33
views
How is the connection between Text Mining, NLP and Tasks like Tokenization, Lemmatization, Stop-word Removal etc.?
I am new to the whole world around Big Data and Text Mining.
It took me a while to understand all the connections and to be able to classify the buzzwords.
But there's one thing I still don't ...
- 27
0
votes
0
answers
13
views
Dictionary of life sciences or medical terminologies
I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
- 285
0
votes
0
answers
46
views
An efficient way to resegment tokenized text into phrases
I have text tokenized on the word level and few lists of phrases stored as tuples.
What would be the most efficient way to resegment (and store) the text into phrases?
For example, a sentence like:
&...
- 1
9
votes
1
answer
8k
views
What tokenizer does OpenAI's GPT3 API use?
I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-...
0
votes
1
answer
354
views
How to perform tokenization for tweets in xlnet?
X_train has only one column that contains all tweets.
...
- 31
1
vote
1
answer
272
views
dealing with HuggingFace's model's tokens
I have a few questions regarding tokenizing word/characters/emojis for different huggingface models.
From my understanding, a model would only perform best during inference if the token of the input ...
- 11
1
vote
1
answer
20
views
Watch list of Tweets with unknown model
I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
5
votes
1
answer
3k
views
Unigram tokenizer: how does it work?
I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it.
I tried to read the ...
- 195
2
votes
1
answer
1k
views
How to i get word embeddings for out of vocabulary words using a transformer model?
When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
- 121
5
votes
2
answers
2k
views
Converting paragraphs into sentences
I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's ...
- 285
0
votes
1
answer
2k
views
BERT uses WordPiece, RoBERTa uses BPE
In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned:
The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
- 117
2
votes
1
answer
402
views
From where does BERT get the tokens it predicts?
When BERT is used for masked language modeling, it masks a token and then tries to predict it.
What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
- 213
1
vote
1
answer
454
views
Can I fine-tune the BERT on a dissimilar/unrelated task?
In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:
"During pre-training, the model is trained on unlabeled data over different pre-training tasks."
I am not sure if I ...
- 117
1
vote
1
answer
216
views
Why BERT tokenizers function differently?
While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer:
...
- 153
-1
votes
1
answer
499
views
How does a neural tokenizer work? [closed]
I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens.
But it is not clear to me how this kind of model should work in terms of the output format. If the ...
- 511
3
votes
1
answer
2k
views
NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?
I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text ...
- 359