14
votes
Accepted
What tokenizer does OpenAI's GPT3 API use?
Tokenizer for GPT-3 is the same as GPT-2:
https://huggingface.co/docs/transformers/model_doc/gpt2#gpt2tokenizerfast
linked via:
https://beta.openai.com/tokenizer
UPDATE March 2023
For newer models, ...
- 256
7
votes
ChatGPT: How to use long texts in prompt?
I didn't find the site you mention that useful - perhaps it is not using the latest GPT model? Or ChatGPT does not yet have very good ability to understand the pdf file I provided it.
Boring answer #1 ...
- 1,074
5
votes
Accepted
Unigram tokenizer: how does it work?
The explanation in the documentation of the Huggingface Transformers library seems more approachable:
Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving ...
- 22.2k
5
votes
What is the difference between TextVectorization and Tokenizer?
Tokenization is the process of splitting a stream of language into individual tokens.
Vectorization is the process of converting string data into a numerical representation.
- 20.2k
4
votes
Accepted
How do NLP tokenizers handle hashtags?
The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using.
Consider this example tweet:
Hi, we need you!
<...
- 1,842
4
votes
Accepted
NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?
Subword tokenization is the norm nowadays in NLP models because:
It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. ...
- 22.2k
4
votes
Accepted
How was the token library constructed for ChatGPT / other GPT systems?
ChatGPT uses byte-pair encoding (BPE) as tokenization strategy. This approach was first proposed in the scientific article Neural Machine Translation of Rare Words with Subword Units. BPE uses subword-...
- 22.2k
3
votes
Accepted
How to customize word division in CountVectorizer?
It's possible if you define CountVectorizer's token_pattern argument.
If you're new to regular expressions, Python's documentation goes over how it deals with ...
- 663
3
votes
Accepted
Converting paragraphs into sentences
Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence ...
- 22.2k
3
votes
Why BERT tokenizers function differently?
Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary ...
- 22.2k
3
votes
3
votes
how to avoid tokenizing w/ sklearn feature extraction
You can change the tokenizer instead of the token pattern. The token pattern is used by the tokenizer. You can set as tokenizer any function that returns a list of string. For example, str.split
<...
3
votes
Accuracy of word and sent tokenize versus custom tokenizers in nltk
Why would we want a custom tokenizer?
Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer. Any toolkit needs to be flexible, and the ability to change ...
- 1,783
3
votes
how do we adapt LLM token embeddings with custom vocab
First, the token vocabulary is extracted from the training data, usually by means of byte-pair encoding (BPE), wordpieces or unigrams. Then, regarding model definition, the first layer in LLMs is the ...
- 22.2k
2
votes
NLP: What are some popular packages for multi-word tokenization?
For your problem i think gensim can be very useful, what can be implemented with Gensim library is phrase detection. It is similar to n-gram, but instead of getting all the n-gram by sliding the ...
- 121
2
votes
NLP: What are some popular packages for multi-word tokenization?
The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation.
One thing you can do is ...
- 133
2
votes
Is it good practice to remove the numeric values from the text data during preprocessing?
To build on Prashant's answer, it will depend on your problem. If you think those values are important to your task, you might try to extract them and tack them onto the end of your data (I'm thinking ...
- 401
2
votes
How to process the hyphenated english words for any nlp problem?
They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known).
For a tfidf ...
- 1,500
2
votes
Converting paragraphs into sentences
There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components
To solve your problem, I see at least three ways to do it.
NTLK
NLTK allows you to add ...
- 5,406
2
votes
Accepted
BERT uses WordPiece, RoBERTa uses BPE
BPE and word pieces are fairly equivalent, with only minimal differences. In practical terms, their main difference is that BPE places the @@ at the end of tokens ...
- 22.2k
2
votes
Accepted
From where does BERT get the tokens it predicts?
There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here.
You can see ...
- 22.2k
2
votes
Accepted
Can I fine-tune the BERT on a dissimilar/unrelated task?
The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked ...
- 22.2k
2
votes
Accepted
How to precompute one sequence in a sequence-pair task when using BERT?
Each token position at each of the attention layers of BERT is computed taking into account all tokens of both sequences. This way, there is not a single element that depends on just the first ...
- 22.2k
2
votes
Accepted
What is the difference between adding words to a tokenizer and training a tokenizer?
First, a clarification: tokenizers receive text and return tokens. These tokens may be words or not. Some tokenizers, for instance, return word pieces (i.e. subwords). This way, a single word may lead ...
- 22.2k
2
votes
Accepted
Smaller embedding size causes lower loss
New Answer
The loss of a text generation task like question generation is normally the average categorical cross-entropy of the output at every time step.
Drastically reducing the number of tokens ...
- 22.2k
2
votes
Accepted
Advantages of different tokenizers for NLP (specifically text generation)
The problem is of text generation. I am assuming you are trying for chatbot etc where input is a natural lanugae and output is a natural language.
Since input is a natural language, all punctuations,...
- 331
2
votes
Accepted
Why is it useful to use different word splitting with different tokenizers?
It's rare to represent sentences as sequences of characters, since most NLP tasks are related to the the semantics of the sentence, which is expressed by the sequence of words. A notable exception: ...
- 24.9k
2
votes
Accepted
What does Codex take as tokens?
NLP neural networks don't use word tokens any more. It's been a while since the norm is using subwords. Usual approaches to define the subword vocabulary are byte-pair encoding (BPE), word pieces or ...
- 22.2k
2
votes
Accepted
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I found this post really helpful for understanding some of the nice properties behind positional embeddings. I'll give a short summary of the relevant portions of the post in my answer, but I highly ...
- 146
2
votes
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
Take a look at the ALiBi paper: https://arxiv.org/abs/2108.12409
For me, the takeaways were:
The sin/cos idea in the "Attention is All You Need" added complexity in the hope it would ...
- 902
Only top scored, non community-wiki answers of a minimum length are eligible