Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Tensorflow text tokenizer incorrect tokenization

I am trying to use TF Tokenizer for a NLP model ...
1
vote
1answer
19 views

How is the connection between Text Mining, NLP and Tasks like Tokenization, Lemmatization, Stop-word Removal etc.?

I am new to the whole world around Big Data and Text Mining. It took me a while to understand all the connections and to be able to classify the buzzwords. But there's one thing I still don't ...
0
votes
0answers
12 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
0
votes
0answers
22 views

An efficient way to resegment tokenized text into phrases

I have text tokenized on the word level and few lists of phrases stored as tuples. What would be the most efficient way to resegment (and store) the text into phrases? For example, a sentence like: &...
0
votes
0answers
50 views

Character Level Embedding in Sentence Classification

I'm working on an NLP task that requires the use of character level embeddings. By using tokenizer library I realized that it tokenizes such as lower integer meant the most frequent character. Is ...
0
votes
0answers
18 views

What tokenizer does OpenAI's GPT3 API use?

I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-...
0
votes
1answer
135 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
1
vote
1answer
73 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
1
vote
1answer
15 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
1
vote
1answer
538 views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
2
votes
1answer
254 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
5
votes
2answers
468 views

Converting paragraphs into sentences

I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's ...
0
votes
1answer
577 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
1
vote
1answer
104 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
1
vote
1answer
214 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
1answer
63 views

Why BERT tokenizers function differently?

While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer: ...
-1
votes
1answer
43 views

How does a neural tokenizer work? [closed]

I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens. But it is not clear to me how this kind of model should work in terms of the output format. If the ...
2
votes
1answer
640 views

NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?

I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text ...
0
votes
0answers
42 views

Tensorflow tokeniser: the maximum number of words to keep

Trying to tokenize the IMDB movie reviews by applying Tensorflow tokenizer. I want to have a maximum 10000-word vocabulary. For unseen words, I use a default token. ...
0
votes
1answer
93 views

Is it good practice to remove the numeric values from the text data during preprocessing?

Im doing preprocessing on a text dataset. I have certain numerics in it like: date(1st July) year(2019) tentative values (3-5 years/ 10+ advantages). unique values (room no 31/ user rank 45) ...
1
vote
3answers
465 views

How to process the hyphenated english words for any nlp problem?

Im doing preprocessing on english text dataset. I encounter hyphenated words like 'well-known'. Will it be useful if I remove the hyphen as special character and treat it as a single word 'wellknown' ...
1
vote
1answer
29 views

create sequence of non dictionary words

I have a few word vectors- recvfrom,sendto,epoll_pwait,recvfrom,sendto,epoll_pwait getuid,recvfrom,writev,getuid,epoll_pwait,getuid Now i want to tokenized them ...
1
vote
0answers
114 views

Why does my char level Keras tokenizer add spaces when converting sequences to texts?

I create a tokenizer with import tf tokenizer = tf.keras.preprocessing.text.Tokenizer(split='', char_level=True, ...) tokenizer.fit_to_texts(...) But when I ...
1
vote
0answers
42 views

CountVectorizer vs HashVectorizer for text

I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work ...
0
votes
3answers
4k views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
0
votes
1answer
78 views

How to Calculate semantic similarity between video captions?

I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences. For example, the captions for one video are as follows: These captions are for the same ...
2
votes
1answer
9k views

Tokenization of data in dataframe in python

I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you. Below are my codes: ...
0
votes
4answers
3k views

how to avoid tokenizing w/ sklearn feature extraction

I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be ...
1
vote
2answers
406 views

Unable to resolve Type error using Tokenizer.tokenize from NLTK

I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution ...
1
vote
1answer
750 views

How can I output tokens from MWE Tokenizer?

How to output the tokens produced using MWE Tokenizer? NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the ...
1
vote
1answer
649 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
1
vote
0answers
206 views

simplifying AND OR Boolean Expression [closed]

My problem is turning a string that looks like this. "a OR (b AND c)" into a OR bc if the expression is like ...
1
vote
0answers
168 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
7
votes
1answer
7k views

Understanding the effect of num_words of Tokenizer in Keras

Consider the following code: ...
2
votes
1answer
2k views
0
votes
1answer
1k views

How do NLP tokenizers handle hashtags?

I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?
5
votes
1answer
1k views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
8
votes
6answers
11k views

NLP: What are some popular packages for multi-word tokenization?

I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are ...