Questions tagged [tokenization]

The tag has no usage guidance.

26 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2 votes
1 answer
755 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
  • 121
1 vote
1 answer
21 views

Is there a tokenizer to tokenize Swift language code in python

...
1 vote
0 answers
100 views

What are the inputs of encoder and decoder layers of transformer architecture?

In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
  • 101
1 vote
1 answer
43 views

How to deal with "Ergänzungsstrichen" and "Bindestrichen" in German NLP?

Problem In German, the phrase "Haupt- und Nebensatz" has exactly the same meaning as "Hauptsatz und Nebensatz". However, when transforming both phrases using e.g. spacy's ...
1 vote
0 answers
19 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
1 vote
1 answer
24 views

How is the connection between Text Mining, NLP and Tasks like Tokenization, Lemmatization, Stop-word Removal etc.?

I am new to the whole world around Big Data and Text Mining. It took me a while to understand all the connections and to be able to classify the buzzwords. But there's one thing I still don't ...
  • 27
1 vote
1 answer
17 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
1 vote
1 answer
30 views

create sequence of non dictionary words

I have a few word vectors- recvfrom,sendto,epoll_pwait,recvfrom,sendto,epoll_pwait getuid,recvfrom,writev,getuid,epoll_pwait,getuid Now i want to tokenized them ...
1 vote
1 answer
234 views

Why does my char level Keras tokenizer add spaces when converting sequences to texts?

I create a tokenizer with import tf tokenizer = tf.keras.preprocessing.text.Tokenizer(split='', char_level=True, ...) tokenizer.fit_to_texts(...) But when I ...
1 vote
0 answers
79 views

CountVectorizer vs HashVectorizer for text

I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work ...
  • 49
1 vote
2 answers
548 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1 vote
0 answers
192 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
0 votes
0 answers
7 views

Looking for an algorithm (preferably rule-based) to segmentize Esperanto words into meaningful word parts?

I am looking for a (rule-based) algorithm to segmentize Esperanto words selecting roots, affixes, and endings. It should also recognize compound words such as 'vaporŝipo' or even 'fivaporŝipegaro'. I ...
0 votes
0 answers
13 views

What does GloVe map with the Tokenizer in embedding?

I have a problem. I have an NLP task that I want to work on. First, I tokenized my text using a tokenizer, then I adjusted the length to be uniform. Then the embedding step is done. Here I use a pre-...
  • 89
0 votes
0 answers
13 views

How can I use the torchtext pad_token tool, or what can I use instead?

I want to test Captum on my nlp model that classifies texts to understand how it classifies them. For that I use the Captum tutorial which uses a pad_token from <...
0 votes
1 answer
15 views

Why GloVe model (by gensim) does not have vectors for numbers 1, 2, ...?

I expected GLoVe to have vectors for numbers. from gensim import downloader as api glove = api.load("glove-twitter-25") glove['1'] This results in ...
  • 101
0 votes
0 answers
74 views

Slow and Fast tokenizer gives different outputs(sentencepiece tokenizer)

When i use T5TokenizerFast(Tokenizer of T5 arcitecture), the output is expected as follows: ['▁', '</s>', '▁Hello', '▁', '<sep>', '</s>'] But ...
  • 101
0 votes
0 answers
46 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
  • 289
0 votes
0 answers
17 views

Any way to make NER tagging with float(2.0) and inferencing with str(2)

One of the NER attribute is tagged with float (3.0, 2.0, ...) while the text file I am trying to inference from are in string format of (3, 2, ...). The Spacy model I used can't pick up the numbers ...
0 votes
2 answers
1k views

Adding a new token to a transformer model without breaking tokenization of subwords

I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
  • 101
0 votes
0 answers
26 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
0 votes
1 answer
13 views

Training NMT models for noisy social media roman text

I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the ...
  • 131
0 votes
0 answers
13 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
  • 285
0 votes
0 answers
39 views

An efficient way to resegment tokenized text into phrases

I have text tokenized on the word level and few lists of phrases stored as tuples. What would be the most efficient way to resegment (and store) the text into phrases? For example, a sentence like: &...
  • 1
0 votes
1 answer
309 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
  • 31
0 votes
1 answer
128 views

How to Calculate semantic similarity between video captions?

I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences. For example, the captions for one video are as follows: These captions are for the same ...