Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
0 votes
0 answers
3 views

Adding words to vocabulary on pre-trained ASR model

I have a pre-trained ASR model but want to add some missing words to the vocabulary. Can I do this or will it invalidate the entire training? Lets say I use the pretrained model: wav2vec2-base-960h ...
28 votes
4 answers
24k views

When to use cosine simlarity over Euclidean similarity

In NLP, people tend to use cosine similarity to measure document/text distances. I want to hear what do people think of the following two scenarios, which to pick, cosine similarity or Euclidean? ...
2 votes
0 answers
10 views

NLP : What is the difference between Authorship Attribution, Authorship Identification and Authorship Recognition?

I have to write my Master's thesis on this topic (I'm in Natural Language Processing) and while I sometimes see these terms used interchangeably other sources seem to emphasize the fact that there are ...
1 vote
1 answer
17 views

Is there a sensible notion of 'character embeddings'?

There are several popular word embeddings available (e.g., Fasttext and GloVe); In short, those embeddings are a tool to encode words along with a sensible notion of semantics attached to those words (...
1 vote
1 answer
137 views

How to train NER LSTM on single sentence level

My documents are only a single sentence long, containing one annotation. Sentences with the same named entity of course are similar, but not context-wise. NER ...
0 votes
1 answer
251 views

'list' object has no attribute 'lower' TfidfVectorizer

I have a dataframe with two text columns and I converted them to a list. I seperated the train and test data as well. But while making a base model TfidfVectorizer throws me an error of 'list' object ...
2 votes
1 answer
510 views

How to find possible subjects for given verb in everyday object domain

I am asking for tools (possibly in NLTK) or papers that talk about the following: e.g. Input: Vase(Subject1) put(verb) Ans I am looking for: flower, water Is there a tool that can output subjects (...
1 vote
2 answers
40 views

How to get sentiment score for a word in a given dataset

I have a sentiment analysis dataset that is labeled in three categories: positive, negative, and neutral. I also have a list of words (mostly nouns), for which I want to calculate the sentiment value, ...
0 votes
0 answers
9 views

Labeling a dataset for sentiment analysis

I was reading articles on sentiment analysis and NLP and there is something I cant quite understand. One of the methods to label a dataset is to use something like textblob with a polarity dictionary ...
0 votes
1 answer
175 views

Loading a Model with weights and optimizers without creating an instance in PyTorch

I recently downloaded Camembert Model to fine-tune it for my purpose. Upon unzipping the file the contents are: Upon loading the model.pt file using pytorch: ...
1 vote
1 answer
50 views

Classification using texts as features

I want to build a classification model to match customers and products. I have a description of each product, and a description of each customer, and the label : ...
0 votes
1 answer
65 views

Optimal input setup for character-level text classification RNN

I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
1 vote
1 answer
570 views

HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00

I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
0 votes
1 answer
43 views

Word-level text generation with word embeddings – outputting a word vector instead of a probability distribution

I am currently researching the topic of text generation for my university project. I decided (ofc) to go with a RNN getting a sequence of tokens as input with a target of predicting the next token ...
9 votes
2 answers
274 views

How to implement hierarchical labeling classification?

I am currently working on the task of eCommerce product name classification, so I have categories and subcategories in product data. I noticed that using subcategories as labels delivers worse results ...
0 votes
0 answers
15 views

Calculate an ambiguity score based on topic models and Hellinger distance

I am trying to calculate some sort of ambiguity score from text based on topic probabilities from a Latent Dirichlet Allocation model and the Hellinger distance between the topic distributions. Let’s ...
0 votes
0 answers
13 views

How to optimize hyperparameters in Bert?

I am using the BERT model in order to classify stereotypes in sentences. I wanted to know if there is a way to automate the optimization of hyperparameters such as 'epochs', 'batchs' or 'learning rate'...
0 votes
0 answers
6 views

How are the embedding and context matrices created and updated in word embedding?

I am struggling to understand how word embedding works, especially how the embedding matrix $W$ and context matrix $W'$ are created/updated. I understand that in the Input we may have a one-hot ...
0 votes
1 answer
99 views

Trying to compress text with NLP

For a university project, I need to send text in Spanish via SMS. As these have a cost, I am trying to compress this text in an inefficient way. This consists of first generating a permutation of ...
28 votes
7 answers
48k views

How to get sentence embedding using BERT?

How to get sentence embedding using BERT? ...
0 votes
0 answers
35 views

Can i use Transformer-XL for text classification task?

I want to use transformer xl for text classification tasks. But I don't know the architect model for the text classification task. I use dense layers with activation softmax for logits output from the ...
0 votes
0 answers
15 views

Dealing with near duplicates using NLP

I have a dataframe like as shown below ...
0 votes
0 answers
7 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
2 votes
1 answer
75 views

Extract details from bibliometrics data

I have set of bibliometrics data (references). I want to extract the author names, title and the name of the conference/journal from it. Since the referencing style used by different papers vary, I am ...
0 votes
1 answer
262 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
1 vote
1 answer
59 views

Supervised learning on sources of information with different importance

I am trying to classify customer support sessions using supervised machine learning. In each customer support session I have 3 bags of information. 1. The title of the customer's complaint 2. ...
1 vote
1 answer
16 views

How to train a machine learning model for named entity recognition

I cannot find any sources about the architectures of machine learning models to solve for NER problems. I vaguely knows it is a multiclass classification problem, but how can we format our input to ...
0 votes
1 answer
152 views

Accessing Flask WS APIs over intranet -

I have 2 scripts - A.py and B.py, and both are Flask apps. A.py renders a web page and acts as my UI taking inputs from user. B.py is hold the main logic and has a web service API being called by A.py....
1 vote
1 answer
247 views

How can I create a "trained" dataset for categorizing news articles?

I am trying to automatically categorize news articles according to their primary topics, i.e. politics, entertainment, sports, business, technology, health, etc. There are some labeled datasets out ...
1 vote
2 answers
583 views

How can I improve the recall of a certain class in a multiclass-classification result

I am working on a multiclass classification which is to assign medical related queries of web search to certain departments of hospital.My classifier is based on the fastText. I found for most ...
-1 votes
0 answers
28 views

How to constrain a dataframe to specific date range?

I have a dataframe that I would like to chunck- or rather run temporal sentiment analysis at different times. I am trying to measure how sentiment changes as part of user identity in extremist social ...
1 vote
1 answer
37 views

Binary document classification using keywords for a very small dataset

I have a set of 150 documents with their assigned binary class. I also have 1000 unlabeled documents. Each document is about the length of a journal paper. Each class has 15 associated keywords. I ...
0 votes
0 answers
9 views

Difference between speech recognition and automatic speech recognition

I was wondering, is there a difference between Speech Recognition and Automatic Speech Recogntion? I have seen both terms used in various papers, and I am not sure whether they are simply used ...
2 votes
1 answer
251 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
0 votes
1 answer
80 views

Keep word2vexc/fasttext model loaded in memory without using API

I have to use Fasttext model to return word embeddings. In test I was calling it through API. Since there are too many words to compute embeddings, API call seems to be expensive. I would like to use ...
2 votes
1 answer
322 views

Integration of NLP and Angular application

I'm doing a small POC in which I've trained my Machine Learning model (Naive Bayes) and is saved in ".pkl" (pickle) format. Now my next task is to develop a web application which asks the user to ...
0 votes
1 answer
73 views

How does an RNN differ from a CBOW model

CBOW: We are trying to predict the next word based on the context (defined as a certain window of words around the target word) RNN can also be used for predicting the next word in a sequence, where ...
1 vote
2 answers
103 views

How best to embed large and noisy documents

I have a large corpus of documents (web pages) collected from various sites of around 10k-30k chars each, I am processing them to extract relevant text as much as possible, but they are never perfect. ...
0 votes
1 answer
48 views

How to compute sentence embedding from word2vec model?

I am new to NLP and I'm trying to perform embedding for a clustering problem. I have created the word2vec model using Python's gensim library, but I am wondering ...
1 vote
0 answers
13 views

Is it possible use cluster analysis on word co-occurrences?

Problem: I am unsure if there is an appropriate clustering method to do the following: I wish to group a list of word co-occurrences into their possible clusters. Context: I have a dataset containing (...
4 votes
1 answer
87 views

Sum vs mean of word-embeddings for sentence similarity

So, say I have the following sentences ["The dog says woof", "a king leads the country", "an apple is red"] I can embed each word using an ...
1 vote
0 answers
6 views

Evaluation of the preprocessing to make a dataset anonymous

I have a very huge dataset from the NLP area and I want to make it anonymous. Is there any way to check if my pre-processing is correct? Generaly, is there any way to evaluate how good is the pre-...
0 votes
0 answers
8 views

NLP based age calculation

I am working on a chatbot with open domain questions using google API & Wikipedia API. Its working few cases like who, what, when kind of questions. when I ask for Age of someone, its collecting ...
0 votes
1 answer
25 views

Optimal clusters for K-means not clear - any ideas?

I have a toy dataset of 10,000 strings of people's names, addresses and birthdays. As a quirk of the data collection process it is highly likely there are duplicate people caused by typos and I am ...
0 votes
0 answers
12 views

How to detect Covariate shift of NLP models?

I have an NLP model, for example, Sentiment Analysis. This model serves in production. I want to detect Data Drift, and specifically Covariate Shift for this model. I saw that Cosine Similarity may ...
0 votes
1 answer
18 views

Word similarity considering special characteristics

I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
0 votes
1 answer
50 views

Start & End Tokens in LSTM when making predictions

I see examples of LSTM sequence to sequence generation models which use start and end tokens for each sequence. I would like to understand when making predictions with this model, if I'd like to make ...
1 vote
2 answers
3k views

Class token in ViT and BERT

I'm trying to understand the architecture of the ViT Paper, and noticed they use a CLASS token like in BERT. To the best of my understanding this token is used to gather knowledge of the entire class, ...
0 votes
0 answers
7 views

Byte-level BPE : Neural Machine Translation with Byte-Level Subwords

For Neural Machine Translation with Byte-Level Subwords , why BBPE outputs are the same for 4K until 32K ?
1 vote
2 answers
182 views

How to use text as an input for a neural network - regression problem? How many likes/claps an article will get

I am trying to predict the number of likes an article or a post will get using a NN. I have a dataframe with ~70,000 rows and 2 columns: "text" (predictor - strings of text) and "likes&...

1
2 3 4 5
47