Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
0 votes
0 answers
3 views

Text similarity for bad-written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
user avatar
0 votes
0 answers
6 views

Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?

I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
user avatar
1 vote
0 answers
11 views

Contextual word embeddings from pretrained word2vec vectors

I would like to create word embeddings that take context into account, so the vector of the word Jaguar [animal] would be different from the word Jaguar [car brand]. As you know, word2vec only gives ...
user avatar
1 vote
1 answer
23 views

Entity Embeddings of email address

I have a set of email address e.g. guptamols@gmail.com, neharaghav@yahoo.com, rkart@gmail.com, squareyards321@ymail.com..... Is it possible to apply ML/Mathematics to generate category (like NER) from ...
user avatar
  • 111
0 votes
0 answers
8 views

Adding words to vocabulary on pre-trained ASR model

I have a pre-trained ASR model but want to add some missing words to the vocabulary. Can I do this or will it invalidate the entire training? Lets say I use the pretrained model: wav2vec2-base-960h ...
user avatar
  • 1
2 votes
0 answers
15 views

NLP : What is the difference between Authorship Attribution, Authorship Identification and Authorship Recognition?

I have to write my Master's thesis on this topic (I'm in Natural Language Processing) and while I sometimes see these terms used interchangeably other sources seem to emphasize the fact that there are ...
user avatar
1 vote
1 answer
22 views

Is there a sensible notion of 'character embeddings'?

There are several popular word embeddings available (e.g., Fasttext and GloVe); In short, those embeddings are a tool to encode words along with a sensible notion of semantics attached to those words (...
user avatar
0 votes
0 answers
9 views

Labeling a dataset for sentiment analysis

I was reading articles on sentiment analysis and NLP and there is something I cant quite understand. One of the methods to label a dataset is to use something like textblob with a polarity dictionary ...
user avatar
  • 101
0 votes
0 answers
15 views

Calculate an ambiguity score based on topic models and Hellinger distance

I am trying to calculate some sort of ambiguity score from text based on topic probabilities from a Latent Dirichlet Allocation model and the Hellinger distance between the topic distributions. Let’s ...
user avatar
0 votes
0 answers
14 views

How to optimize hyperparameters in Bert?

I am using the BERT model in order to classify stereotypes in sentences. I wanted to know if there is a way to automate the optimization of hyperparameters such as 'epochs', 'batchs' or 'learning rate'...
user avatar
  • 320
0 votes
0 answers
7 views

How are the embedding and context matrices created and updated in word embedding?

I am struggling to understand how word embedding works, especially how the embedding matrix $W$ and context matrix $W'$ are created/updated. I understand that in the Input we may have a one-hot ...
user avatar
0 votes
0 answers
37 views

Can i use Transformer-XL for text classification task?

I want to use transformer xl for text classification tasks. But I don't know the architect model for the text classification task. I use dense layers with activation softmax for logits output from the ...
user avatar
  • 1
0 votes
0 answers
16 views

Dealing with near duplicates using NLP

I have a dataframe like as shown below ...
user avatar
  • 2,129
0 votes
0 answers
8 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
user avatar
  • 257
1 vote
1 answer
16 views

How to train a machine learning model for named entity recognition

I cannot find any sources about the architectures of machine learning models to solve for NER problems. I vaguely knows it is a multiclass classification problem, but how can we format our input to ...
user avatar
  • 359
0 votes
0 answers
10 views

Difference between speech recognition and automatic speech recognition

I was wondering, is there a difference between Speech Recognition and Automatic Speech Recogntion? I have seen both terms used in various papers, and I am not sure whether they are simply used ...
user avatar
  • 1
1 vote
0 answers
13 views

Is it possible use cluster analysis on word co-occurrences?

Problem: I am unsure if there is an appropriate clustering method to do the following: I wish to group a list of word co-occurrences into their possible clusters. Context: I have a dataset containing (...
user avatar
4 votes
1 answer
92 views

Sum vs mean of word-embeddings for sentence similarity

So, say I have the following sentences ["The dog says woof", "a king leads the country", "an apple is red"] I can embed each word using an ...
user avatar
1 vote
0 answers
6 views

Evaluation of the preprocessing to make a dataset anonymous

I have a very huge dataset from the NLP area and I want to make it anonymous. Is there any way to check if my pre-processing is correct? Generaly, is there any way to evaluate how good is the pre-...
user avatar
0 votes
0 answers
8 views

NLP based age calculation

I am working on a chatbot with open domain questions using google API & Wikipedia API. Its working few cases like who, what, when kind of questions. when I ask for Age of someone, its collecting ...
user avatar
0 votes
0 answers
12 views

How to detect Covariate shift of NLP models?

I have an NLP model, for example, Sentiment Analysis. This model serves in production. I want to detect Data Drift, and specifically Covariate Shift for this model. I saw that Cosine Similarity may ...
user avatar
0 votes
1 answer
18 views

Word similarity considering special characteristics

I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
user avatar
  • 1
-1 votes
0 answers
28 views

How to constrain a dataframe to specific date range?

I have a dataframe that I would like to chunck- or rather run temporal sentiment analysis at different times. I am trying to measure how sentiment changes as part of user identity in extremist social ...
user avatar
0 votes
1 answer
26 views

Optimal clusters for K-means not clear - any ideas?

I have a toy dataset of 10,000 strings of people's names, addresses and birthdays. As a quirk of the data collection process it is highly likely there are duplicate people caused by typos and I am ...
user avatar
  • 187
0 votes
0 answers
7 views

Byte-level BPE : Neural Machine Translation with Byte-Level Subwords

For Neural Machine Translation with Byte-Level Subwords , why BBPE outputs are the same for 4K until 32K ?
user avatar
  • 283
0 votes
0 answers
25 views

Language Detection using pycld2

I am trying to use the pycld2 package to detect multiple languages in text. This package provides Python bindings for the Compact Language Detect 2 (CLD2) This is the example I am testing out: ...
user avatar
  • 11
0 votes
2 answers
22 views

Suggestions for a multi-class text classification model with a large number of classes?

I was working on a text classification problem where I currently have around 40-45 different labels. The input is a text sentence with a keyword. For e.g. ...
user avatar
0 votes
1 answer
9 views

Ideal Windows Size in Pk Evaluation Metric

I am very new to nlp. I am doing a text segmentation task and for evaluating my model I need to calculate Pk and Windiff scores. My question is what is the ideal value for window size (k) for Pk score ...
user avatar
  • 1
1 vote
0 answers
32 views

getting actual concepts value instead of its URI in ontology

I am using owl ontology for semantic analysis in emotional sentiment analysis project , I am trying to navigate the ontology to check a concepts and its relation , my ontology has classes like this : <...
user avatar
2 votes
0 answers
13 views

Looking for a generalized (extended) lemmatizer

Whenever I lemmatize a compound word in English or German, I obtain a result that ignores the compound structure, e.g. for 'sidekicks' the NLTK WordNet lemmatizer returns 'sidekick', for '...
user avatar
0 votes
0 answers
17 views

Why (or how) does a Keras model skip Stemming or Lemmatization steps?

This Keras article / tutorial here does perform text standardization i.e removing HTML elements, punctuation, etc. from the text dataset, however, there is a distinct lack of any stemming or ...
user avatar
0 votes
0 answers
7 views

How to generate a sentence around words in Keras?

I know that how to generate next word in keras with lstm but how to predict previous word for example If you have two words like "car" and "running" then It should generate "...
user avatar
  • 1
0 votes
0 answers
40 views

Are the word of women and men different when expressing their views on the same subject?

My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting ...
user avatar
  • 9
1 vote
0 answers
17 views

sentence type classification

I want to classify the sentences in my dataset as declarative, interrogative, imperative and ...
user avatar
1 vote
1 answer
33 views

How to perform Grid Search on NLP CRF model

I am trying to perform hyperparameter tuning on sklearn_crfsuite.CRF model. When I try to execute below code, it doesn't give any exception but it probably fails to perform fit. And due to which, if I ...
user avatar
  • 111
0 votes
1 answer
16 views

How do i generate text from ids in Torchtext's sentencepiece_numericalizer?

The torchtext sentencepiece_numericalizer() outputs a generator with indices SentencePiece model corresponding to token in the input sentence. From the generator, I ...
user avatar
  • 17
0 votes
0 answers
5 views

Tagging short strings based on position, case, word frequency and so on

Most of the NLP stuff I've been looking at does NER given a long blob of text (e.g., a news article). I am curious what the best method is when you have millions of short strings, say for example ...
user avatar
0 votes
0 answers
5 views

predictive effect in the classification made according to the comments in different fields

I want to do a classification through comments categorized in 4 areas(X,Y,Z,M). Categorizing the product as good or bad based on the comments in the fields X, Y, Z, M. How can I follow a path to see ...
user avatar
  • 9
0 votes
0 answers
18 views

Models for key value pair extraction in NLP

There are a range of models, articles and information available which extract keywords from unstructured documents. Examples are Spacy, POS tagging, NER, entity extraction etc What I am looking for is ...
user avatar
0 votes
0 answers
12 views

Looking for NLP datasets with small vocabularies

I'm looking for NLP datasets/corpora with small vocabulary sizes -- less than 5K unique words, but smaller still is better. I've tried looking for e.g. datasets of children's books but I've not found ...
user avatar
  • 1
0 votes
0 answers
20 views

Can I use MLM method to fine tune my BERT model, if it was initially trained with natural language inference method?

I am using BERT model for sentence similarity task. However my dataset with sentence is very specific and I want to fine tune my model on it first. My dataset is unlabelled. And BERT model that I want ...
user avatar
  • 113
0 votes
0 answers
23 views

Should I pretrain my BERT model on specific dataset if it has only one class of labels?

I want to use BERT model for sentences similarity measuring task. I know that BERT models were trained with natural language inference architecture with dataset with labels neutral, entailment, ...
user avatar
  • 113
0 votes
0 answers
17 views

While training BERT variant, getting IndexError: index out of range in self

While training XLMRobertaForSequenceClassification: ...
user avatar
  • 137
-1 votes
0 answers
15 views

Error using Spacy in NLP

I am getting this error, please can i resolve this error using the spacy library?
user avatar
  • 9
1 vote
0 answers
11 views

NLP work related to distinguishing scenes in a story

Say I have a story or novel with multiple scenes. Are there any NLP work/techniques that allow me to know when the author switches from one scene to another? Searching things like "context", ...
user avatar
  • 111
2 votes
1 answer
30 views

how to deal with large numbers of unlabelled target dataset?

I have dataset of 5000 jobs descriptions out of which only 200 jobs are labelled with required English level score range between 0 to 9 and I want to predict remaining 4800 jobs required English level ...
user avatar
1 vote
0 answers
15 views

Guide to Natural language Prompt programming for few-shot learning of Pretrained Language Models

I'm currently working on a project with the goal of producing AI content in the space of a content generation like blog writing, Instagram caption generation etc. Found the in-context few-shot ...
user avatar
1 vote
0 answers
9 views

Question Categorization Dataset(s)

I'm looking for a dataset or group of datasets I could combine that would contain numerous examples of the following types of question categories. free response (ex: What is the capital of Portugal?) ...
user avatar
-1 votes
0 answers
23 views

How to decide if semantic information is important when working on a text/sentiment analysis?

How does semantic information influence the model performance?
user avatar
  • 9
1 vote
0 answers
16 views

Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?

I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the ...
user avatar
  • 11

1
2 3 4 5
47