Skip to main content
Share Your Experience: Take the 2024 Developer Survey
14 votes
Accepted

What tokenizer does OpenAI's GPT3 API use?

Tokenizer for GPT-3 is the same as GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#gpt2tokenizerfast linked via: https://beta.openai.com/tokenizer UPDATE March 2023 For newer models, ...
pjama's user avatar
  • 256
7 votes

ChatGPT: How to use long texts in prompt?

I didn't find the site you mention that useful - perhaps it is not using the latest GPT model? Or ChatGPT does not yet have very good ability to understand the pdf file I provided it. Boring answer #1 ...
Valentas's user avatar
  • 1,209
5 votes

What is the difference between TextVectorization and Tokenizer?

Tokenization is the process of splitting a stream of language into individual tokens. Vectorization is the process of converting string data into a numerical representation.
Brian Spiering's user avatar
5 votes
Accepted

Unigram tokenizer: how does it work?

The explanation in the documentation of the Huggingface Transformers library seems more approachable: Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving ...
noe's user avatar
  • 26.9k
5 votes
Accepted

NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?

Subword tokenization is the norm nowadays in NLP models because: It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. ...
noe's user avatar
  • 26.9k
4 votes

Why BERT tokenizers function differently?

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary ...
noe's user avatar
  • 26.9k
4 votes
Accepted

How do NLP tokenizers handle hashtags?

The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using. Consider this example tweet: Hi, we need you! <...
TitoOrt's user avatar
  • 1,872
4 votes
Accepted

How was the token library constructed for ChatGPT / other GPT systems?

ChatGPT uses byte-pair encoding (BPE) as tokenization strategy. This approach was first proposed in the scientific article Neural Machine Translation of Rare Words with Subword Units. BPE uses subword-...
noe's user avatar
  • 26.9k
3 votes
Accepted

Converting paragraphs into sentences

Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence ...
noe's user avatar
  • 26.9k
3 votes

Tokenization of data in dataframe in python

You can try with this: ...
Carlos Mougan's user avatar
3 votes

how to avoid tokenizing w/ sklearn feature extraction

You can change the tokenizer instead of the token pattern. The token pattern is used by the tokenizer. You can set as tokenizer any function that returns a list of string. For example, str.split <...
Hernan C. Vazquez's user avatar
3 votes
Accepted

How to customize word division in CountVectorizer?

It's possible if you define CountVectorizer's token_pattern argument. If you're new to regular expressions, Python's documentation goes over how it deals with ...
Tom M.'s user avatar
  • 671
3 votes

Accuracy of word and sent tokenize versus custom tokenizers in nltk

Why would we want a custom tokenizer? Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer. Any toolkit needs to be flexible, and the ability to change ...
Stephen Rauch's user avatar
  • 1,783
3 votes

how do we adapt LLM token embeddings with custom vocab

First, the token vocabulary is extracted from the training data, usually by means of byte-pair encoding (BPE), wordpieces or unigrams. Then, regarding model definition, the first layer in LLMs is the ...
noe's user avatar
  • 26.9k
2 votes

NLP: What are some popular packages for multi-word tokenization?

For your problem i think gensim can be very useful, what can be implemented with Gensim library is phrase detection. It is similar to n-gram, but instead of getting all the n-gram by sliding the ...
Qaisar Rajput's user avatar
2 votes

NLP: What are some popular packages for multi-word tokenization?

The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation. One thing you can do is ...
David Batista's user avatar
2 votes

Understanding the effect of num_words of Tokenizer in Keras

The problem is with the way things are documented. Check this link: https://stackoverflow.com/questions/46202519/keras-tokenizer-num-words-doesnt-seem-to-work
Prince's user avatar
  • 21
2 votes

Is it good practice to remove the numeric values from the text data during preprocessing?

To build on Prashant's answer, it will depend on your problem. If you think those values are important to your task, you might try to extract them and tack them onto the end of your data (I'm thinking ...
rabbit's user avatar
  • 421
2 votes

How to process the hyphenated english words for any nlp problem?

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known). For a tfidf ...
Nicholas James Bailey's user avatar
2 votes

Converting paragraphs into sentences

There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components To solve your problem, I see at least three ways to do it. NTLK NLTK allows you to add ...
Valentin Calomme's user avatar
2 votes
Accepted

BERT uses WordPiece, RoBERTa uses BPE

BPE and word pieces are fairly equivalent, with only minimal differences. In practical terms, their main difference is that BPE places the @@ at the end of tokens ...
noe's user avatar
  • 26.9k
2 votes
Accepted

From where does BERT get the tokens it predicts?

There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here. You can see ...
noe's user avatar
  • 26.9k
2 votes
Accepted

Can I fine-tune the BERT on a dissimilar/unrelated task?

The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked ...
noe's user avatar
  • 26.9k
2 votes
Accepted

How to precompute one sequence in a sequence-pair task when using BERT?

Each token position at each of the attention layers of BERT is computed taking into account all tokens of both sequences. This way, there is not a single element that depends on just the first ...
noe's user avatar
  • 26.9k
2 votes
Accepted

What is the difference between adding words to a tokenizer and training a tokenizer?

First, a clarification: tokenizers receive text and return tokens. These tokens may be words or not. Some tokenizers, for instance, return word pieces (i.e. subwords). This way, a single word may lead ...
noe's user avatar
  • 26.9k
2 votes
Accepted

Smaller embedding size causes lower loss

New Answer The loss of a text generation task like question generation is normally the average categorical cross-entropy of the output at every time step. Drastically reducing the number of tokens ...
noe's user avatar
  • 26.9k
2 votes
Accepted

Advantages of different tokenizers for NLP (specifically text generation)

The problem is of text generation. I am assuming you are trying for chatbot etc where input is a natural lanugae and output is a natural language. Since input is a natural language, all punctuations,...
amol goel's user avatar
  • 341
2 votes
Accepted

Why is it useful to use different word splitting with different tokenizers?

It's rare to represent sentences as sequences of characters, since most NLP tasks are related to the the semantics of the sentence, which is expressed by the sequence of words. A notable exception: ...
Erwan's user avatar
  • 25.5k
2 votes
Accepted

What does Codex take as tokens?

NLP neural networks don't use word tokens any more. It's been a while since the norm is using subwords. Usual approaches to define the subword vocabulary are byte-pair encoding (BPE), word pieces or ...
noe's user avatar
  • 26.9k
2 votes
Accepted

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I found this post really helpful for understanding some of the nice properties behind positional embeddings. I'll give a short summary of the relevant portions of the post in my answer, but I highly ...
Alexander Wan's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible