Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
21 views

How can I know the expected inputs to a huggingface Trainer?

I'n a Colab notebook by huggingface here they print the raw dataset keys as such: ...
Ameen Izhac's user avatar
1 vote
0 answers
13 views

Which model and classification algorithms should I use for deobfuscating/mapping symbols in source code?

I have a source code for an application that recently started undergoing obfuscation. Each new version alters the names of symbols, shuffles the order of classes, and employs other strategies. However,...
emi's user avatar
  • 111
0 votes
0 answers
7 views

Strategies for Encoding Large Datasets in Symbolic Music Generation for BERT-type Model

I am creating a BERT-type model for symbolic music generation. An observation of my database is a musical piece. Actually, is a "viewpoint" of the piece: ...
Kikolo's user avatar
  • 101
0 votes
0 answers
17 views

Questions about sentencepiece tokenizer

The SentencePiece original paper manuscript is as vague as it can get about the implementation of the algorithm. The paper describes the problems with a BPE or Unigram tokenizer and then claims that ...
figs_and_nuts's user avatar
1 vote
1 answer
39 views

Understanding processors in huggingface tokenizer library

tl;dr What are the :0 and :1 in the following huggingface processors reference usage given on their page: ...
figs_and_nuts's user avatar
0 votes
0 answers
11 views

How are the words defined in the sentencepiece algorithm?

I am not able to understand how the sentencepiece algorithm solves the problem of handling the languages without a clear-cut concept of words My exact confusion is: It is mentioned that one of the ...
figs_and_nuts's user avatar
0 votes
0 answers
8 views

How to pre-processed text to keep dashed words?

I'm working on compiling reviews for movies and analysing it in Orange. I've found that words like "r-rated" get converted to "r" and "rated". I've messed around a bit ...
Rasmus's user avatar
  • 1
0 votes
0 answers
100 views

Sentencepiece Tokenizer training from scrath

To train BPE model on sentencepiece as per given Usage instructions As, it is mentioned in the instructions that --input: one-sentence-per-line raw corpus file. ...
Vinay Sharma's user avatar
1 vote
1 answer
76 views

Why is the sprase categorical accuracy decreasing every epoch and predictions are always NaN?

Problem Summary My model is built and compiled properly but gets the NaN validation loss on all epochs. The training set accuracy is also infinitesimally small and keeps decreasing. I couldn't find a ...
Joachim Rives's user avatar
0 votes
0 answers
56 views

How to improve GPT2 tokenizer trained from scratch?

I trained a GPT2 Tokenizer on Hindi dataset of size 170 MB from scratch and saved it as new_tokenizer. When I tried the new_tokenizer on a Hindi sentence ...
Vinay Sharma's user avatar
0 votes
0 answers
75 views

Help understanding working of KeyBERT for keyphrase extraction

I'm fairly new to reading and understanding research papers, so I wanted to get a second opinion on whether my understanding of KeyBERT was correct. Here is a high level overview of my understanding ...
Prithvi's user avatar
  • 33
1 vote
2 answers
2k views

how do we adapt LLM token embeddings with custom vocab

Hi im just getting started with understanding transformer based models and I am not able to find how the token embeddings are arrived at?. there are multiple tokenization approaches and multiple ...
dasman's user avatar
  • 111
2 votes
2 answers
719 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
2 votes
1 answer
132 views

How was the token library constructed for ChatGPT / other GPT systems?

I have found literally hundreds of articles on Google with titles like 'What are tokens and how to use them,' but haven't been able to find any information at all on how the token libraries themselves ...
Sinnombre's user avatar
  • 153
1 vote
1 answer
109 views

What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?

I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
Austin Capobianco's user avatar
4 votes
2 answers
17k views

ChatGPT: How to use long texts in prompt?

I like the website chatpdf.com a lot. You can upload a PDF file and then discuss the textual content of the file with the file "itself". It uses ChatGPT. I would like to program something ...
meyer_mit_ai's user avatar
0 votes
1 answer
183 views

What does Codex take as tokens?

The typical default for neural networks in natural language processing has been to take words as tokens. OpenAI Codex is based on GPT-3, but also deals with source code. For source code in general, ...
rwallace's user avatar
  • 139
0 votes
0 answers
30 views

Does order matter in this causal language model?

Say you've implemented a causal language model like so: ...
Manny's user avatar
  • 1
2 votes
1 answer
232 views

Does GPT-3 remember data from prompts used to fine tune it?

I am trying to fine tune a model using OpenAI's fine tuning API. I am passing bodies of text (for example, news paper articles) as prompts and the data I want from it as completions. Let us consider ...
Kushal Mohnot's user avatar
1 vote
2 answers
43 views

Why is it useful to use different word splitting with different tokenizers?

I have a problem. I have a NLP classification problem. There are different methods to decompose sentences into tokens, for example in whole words or in characters. Then there are different tokenizers ...
Test's user avatar
  • 89
0 votes
1 answer
112 views

Advantages of different tokenizers for NLP (specifically text generation)

What are the advantages of using different tokenizers? For example, let's take the sentence: "In Düsseldorf I took my hat off. But I can't put it back on." The treebank tokenizer yields: &...
postnubilaphoebus's user avatar
0 votes
1 answer
69 views

What counts as a token for bpemb's encode_ids_with_eos()

I have probelms understanding bpemb's encode_ids_with_eos() or similar. When I run the following code i get none-word like segmentations (rather syllalbus based or ...
Piskator's user avatar
  • 125
0 votes
1 answer
124 views

Why GloVe model (by gensim) does not have vectors for numbers 1, 2, ...?

I expected GLoVe to have vectors for numbers. from gensim import downloader as api glove = api.load("glove-twitter-25") glove['1'] This results in ...
Aidis's user avatar
  • 101
1 vote
1 answer
50 views

Is there a tokenizer to tokenize Swift language code in python

...
Shamsudeen McHalwai's user avatar
0 votes
0 answers
496 views

Slow and Fast tokenizer gives different outputs(sentencepiece tokenizer)

When i use T5TokenizerFast(Tokenizer of T5 arcitecture), the output is expected as follows: ['▁', '</s>', '▁Hello', '▁', '<sep>', '</s>'] But ...
canP's user avatar
  • 101
0 votes
1 answer
188 views

Smaller embedding size causes lower loss

When I convert my multilingual transformer model to a single lingual transformer model (got my languages embedding from the multilingual transformer and deleted other embeddings, decreased dimensions ...
canP's user avatar
  • 101
0 votes
1 answer
57 views

What is the difference between adding words to a tokenizer and training a tokenizer?

The title says it all. I was researching this question but couldn't find something useful. What is the difference between adding words to a tokenizer and training a tokenizer?
canP's user avatar
  • 101
0 votes
1 answer
38 views

What is the effect of the tokens?

What is the effect of the tokens that the model has if model A has 1B tokens and the other model has 12B tokens? Will that have an effect on the performance?
Lei's user avatar
  • 57
1 vote
0 answers
470 views

What are the inputs of encoder and decoder layers of transformer architecture?

In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
canP's user avatar
  • 101
1 vote
1 answer
113 views

How to deal with "Ergänzungsstrichen" and "Bindestrichen" in German NLP?

Problem In German, the phrase "Haupt- und Nebensatz" has exactly the same meaning as "Hauptsatz und Nebensatz". However, when transforming both phrases using e.g. spacy's ...
gebbissimo's user avatar
1 vote
1 answer
104 views

does ValueError: 'rat' is not in list means not exist in tokenizer

Does this error means that the word doesn't exist in the tokenizer return sent.split(" ").index(word) ValueError: 'rat' is not in list the code sequences ...
Begnnier's user avatar
0 votes
2 answers
549 views

How does Keras Tokenizer choose tokens given a sentence?

I tried to find the answer to this question but I can't find anything, so I ask here: How does Keras Tokenizer choose tokens given a sentence of words ? To be more precise with what I want to know, ...
HelpNeederStudent's user avatar
1 vote
1 answer
147 views

How to precompute one sequence in a sequence-pair task when using BERT?

BERT uses separator tokens ([SEP]) to input two sequences for a sequence-pair task. If I understand the BERT architecture correctly, attention is applied to all inputs thus coupling the two sequences ...
Just van der Veeken's user avatar
3 votes
1 answer
2k views

What is the difference between TextVectorization and Tokenizer?

What is the difference between the layers.TextVectorization() and ...
Pritam Sinha's user avatar
1 vote
2 answers
4k views

Adding a new token to a transformer model without breaking tokenization of subwords

I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
Jigsaw's user avatar
  • 121
1 vote
0 answers
24 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
hoshii_tomato's user avatar
0 votes
0 answers
66 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
Sakshi Maurya's user avatar
0 votes
1 answer
16 views

Training NMT models for noisy social media roman text

I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the ...
Gokul NC's user avatar
  • 131
1 vote
1 answer
38 views

How is the connection between Text Mining, NLP and Tasks like Tokenization, Lemmatization, Stop-word Removal etc.?

I am new to the whole world around Big Data and Text Mining. It took me a while to understand all the connections and to be able to classify the buzzwords. But there's one thing I still don't ...
Loretta's user avatar
  • 37
0 votes
0 answers
15 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
Van Peer's user avatar
  • 285
0 votes
0 answers
50 views

An efficient way to resegment tokenized text into phrases

I have text tokenized on the word level and few lists of phrases stored as tuples. What would be the most efficient way to resegment (and store) the text into phrases? For example, a sentence like: &...
Karvin's user avatar
  • 1
9 votes
1 answer
9k views

What tokenizer does OpenAI's GPT3 API use?

I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-...
Herman Autore's user avatar
0 votes
1 answer
364 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
Mathew's user avatar
  • 31
1 vote
1 answer
279 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
user113789's user avatar
1 vote
1 answer
20 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
cavalierstyles's user avatar
5 votes
1 answer
4k views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
Johncowk's user avatar
  • 195
3 votes
1 answer
1k views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
cerofrais's user avatar
  • 131
5 votes
2 answers
2k views

Converting paragraphs into sentences

I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's ...
Van Peer's user avatar
  • 285
0 votes
1 answer
2k views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
Adel's user avatar
  • 127
2 votes
1 answer
440 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
Nick Koprowicz's user avatar