Questions tagged [nlp]
Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.
2,568
questions
0
votes
1
answer
12
views
How do GPT models go from token probabilities to textual outputs?
Suppose GPT-2 or GPT-3 is trying to generate the next token, and it has a probability distribution (after applying softmax to some output logits) for the different possible next tokens. How does it ...
- 101
0
votes
1
answer
12
views
How many parameters does the vanilla Transformer have?
The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
- 163
0
votes
0
answers
6
views
Feature selection in high-dimensional datasets with sparse features
What are the most effective techniques for feature selection in high-dimensional datasets with sparse features in the field of natural language processing?
- 101
1
vote
1
answer
15
views
Why the label is not explicitly involved in the loss function of skip-gram?
I am recently learning word embedding myself. When learning skip-gram from the paper https://arxiv.org/pdf/1310.4546.pdf[Distributed Representations of Words and Phrases and their Compositionality], I ...
- 11
0
votes
0
answers
7
views
For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?
From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
- 1
1
vote
1
answer
34
views
Convert cosine similarity to probability
In natural language processing, the cosine similarity is often used to compute the similarity between two words. It is bounded between [-1, 1]. Supposedly, 1 means complete similarity, -1 means ...
0
votes
1
answer
18
views
GPT-2 architecture question
I am currently working on a NLP model that compares two comments and determines which one would be more popular. I have already came up with an architecture - it will be based on GPT-2. But now I am ...
0
votes
0
answers
14
views
Subjectivity Classification with BERT and Word2Vec
I am new to NLP and working on a final-year project to classify if a sentence is written from objective or subjective point of view, using BERT with Word2Vec.
The datasets I found for this project are ...
0
votes
0
answers
15
views
How to know which model is good for my usecase?
I need to summarize and answer some questions about articles. For this, I will use language models. I see that GPT-3 works amazingly good for my use case. But, there's also Alpaca.
They both look fine ...
- 101
1
vote
1
answer
24
views
Using BERT to extract a list of words and phrases from documents
I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data.
For example my list of words and ...
- 385
1
vote
1
answer
12
views
Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?
I am using BERT to do multiclass text classification. The number of output classes I have to predict from is: 116 and there is high degree of class imbalance that I see.
We have the following kind of ...
0
votes
0
answers
6
views
Is there any concern for a pretrained model to overfitting to a fine-tuning task that has overlapping pretraining and training data?
Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any ...
- 1
0
votes
0
answers
7
views
Algorithm of lda2vec in NLP
I was going through lda2vec and was confused on some of the concepts.It is a combination of LDA and word2vec.Word2vec is used to learn dense word vectors and LDA is used to learn the probability ...
- 1
0
votes
0
answers
9
views
Creating a neural network for classification that take each embedded word in each sentence as input
(The title was difficult to phrase - please suggest another title if you can)
I have a classification problem with 60 classes, and some (very) short sentences (bank transactions). Most of the ...
- 440
0
votes
1
answer
21
views
how to find the specific pdf files with os.path.join()
Can anyone help me move the files containing the specific student number based on the number in the CSV file to another folder? The code ran without any error, but it does not do anything.
Is the code ...
- 1
1
vote
0
answers
15
views
A French version of Rebel
Is there an end-to-end trained transformer like Rebel for french data?
Rebel can extract entities and relations from text, yet as far as I know, it works only with english texts.
Is there any other ...
- 11
0
votes
0
answers
11
views
Understanding gradient of skip gram
I am trying to understand gradient calculation for skip gram with softmax output and cross entropy loss.
I am referring these articles: 1, 2, 3.
The all calculate the error as follows:
$$E=-\sum_{c=1}...
- 99
0
votes
0
answers
12
views
Stop words removal vs word subsampling for word2vec
I was exploring skip gram optimization from this article. In this section, it explains sub sampling to sample frequent words as follows:
Probability of keeping the word is: $$(w_i)=\left(\sqrt{\frac{...
- 139
0
votes
1
answer
28
views
SBERT Embeddings from Conversations
I have a dataset consisting of text-based conversations between two humans. One conversation has on average 20 turns and can look as follows:
...
- 121
3
votes
1
answer
24
views
Is there a reference dataset for contextual similarity?
I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but ...
- 241
1
vote
1
answer
27
views
How does BERT work for Aspect-Based sentiment analysis?
I have recently used a package to perform Aspect-Based Sentiment Analysis (ABSA) through a BERT model.
Briefly, the model takes two inputs:
words that constitute the aspects
a sentence on which we ...
0
votes
0
answers
24
views
How to finetune a closed generative huggingface model?
I want to finetune a huggingface pretrained model on our internal documentation in a way it stats answering related questions. I could not find the adequate tutorial.
- 121
2
votes
2
answers
25
views
Should I open abbreviations/acronyms in the text data, when training transformer model?
I am currently training a transformer model on text data. Is it a good practise to open abbreviations/acronyms in the text data? I did not dins any tips or recommendations about it on internet.
- 183
0
votes
0
answers
13
views
Finetuning fasttext with unlabeled text corpus
I am training a classifier which is supposed to take the name of a product as input. For this purpose I want to finetune a pre-existing fasttext model on my article names.
My code looks like this
<...
1
vote
1
answer
27
views
Understand the interpretability of word embeddings
When reading the Tensorflow tutorial for Word Embeddings, I found two notes that confuse me:
Note: Experimentally, you may be able to produce more interpretable embeddings by using a simpler model. ...
- 13
0
votes
0
answers
9
views
How to calibrate autoregressive model?
TL;DR What metric, and how to calibrate autoregressive language (deep learning) model?
Background
Usually there are several popular approach to calibrate a non-autoregressive model such as isotonic ...
- 101
0
votes
1
answer
48
views
What are the advantages of autoregressive over seq2seq?
Why are recent dialog agents, such as ChatGPT, BlenderBot3, and Sparrow, based on the decoder architecture instead of the encoder-decoder architecture?
I know the difference between the attention of ...
- 3
0
votes
0
answers
16
views
Spacy POS tag for quoted text
I want Spacy to treat quoted text as e.g. a NOUN or PROPN so that it forms a compound with the adjacent word. For example in the following text I want the text "drink me" to be attached to ...
1
vote
1
answer
41
views
How do we evaluate the outputs of text generation models?
Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
- 111
0
votes
0
answers
22
views
Best string similarity metric not considering word order
I'm sorry if the title is misleading, but I didn't really know how to explain what I am searching for.
I have a dataset containing two columns representing names and surnames of a bunch of people. ...
- 1
0
votes
0
answers
20
views
Does order matter in this causal language model?
Say you've implemented a causal language model like so:
...
- 1
0
votes
0
answers
19
views
Can this task for phrases be called lemmatization?
I want to 'lemmatize' phrases to dictionary entries. For instance, the following collocates can be standardized to the idiom in the aforementioned link ...
- 436
2
votes
2
answers
87
views
Group unstructured chat logs into conversations
I am new to ML/AI/NLP and am interested in tackling the following problem. I have a database of chat logs from a Discord server. The database contains the following labeled data: ...
- 21
0
votes
2
answers
44
views
Data Preparation for next word prediction
In most places, I have seen that when preparing the training data and label for next-word prediction from the corpus one uses a fixed window size say of length 4, and then scans the subsequences of ...
- 101
0
votes
0
answers
27
views
is the distilRoberta transformer model overfitting or underfitting?
I am a bit new to ML, below are the results after I fine tuned distilRoBERTa using HuggingFace Trainer. I cant tell if my model is over-fitting, under-fitting or ok? I ran 7 epochs.
I think its ...
- 1
0
votes
1
answer
18
views
Why custom training a Spacy model runs only the Initializing pipeline but the Training pipeline is not running?
I am training a custom NER model with Spacy version 3.5.0 using some dummy data. My entire code and dummy data is given below. This is exact same code give in the 2nd half of this link. The code is ...
- 185
1
vote
1
answer
34
views
How to extract values from unstructured text
I'm implementing a tool which should extract values of interest from unstructured text entries. The data set is several hundred thousands of medical entries. Each entry is relatively short (around 100 ...
- 111
0
votes
0
answers
9
views
How to extract a relevant paragraph/content from a multiple documents as per the question asked by the user?
I have a multiple similar text file in which part of the answer resides as per my questions and I want to only extract that paragraph from the most similar documents. I have used FAISS for searching ...
0
votes
0
answers
12
views
Using NLP output for subsequent model?
I use SpaCy to output a vectorized array of my text field. I'm having issues plugging this output into my random forest and could use some guidance. I label encoded other fields so my pandas dataframe ...
- 11
0
votes
0
answers
47
views
Does GPT-3 remember data from prompts used to fine tune it?
I am trying to fine tune a model using OpenAI's fine tuning API. I am passing bodies of text (for example, news paper articles) as prompts and the data I want from it as completions.
Let us consider ...
0
votes
0
answers
29
views
How to fine-tune a large language model for translation in a multi-dataset setting?
The Problem
We need to translate from language N to language C. If it helps, N is a natural ...
0
votes
1
answer
24
views
Alternatives to Toronto Book Corpus
As the toronto book corpus is no longer available (or rather, only in lowercase), I am looking for an alternative dataset of comparable language variety, quality, and size.
Any suggestions? The ...
0
votes
1
answer
31
views
What loss function to use for predicting discrete output sequence given a discrete input sequence?
I am working on sequence-to-sequence tasks where the input is an n-length sequence of discrete values from a finite set S (say ...
0
votes
1
answer
25
views
Word2vec CBOW model with negative sampling
From this article:
In vanilla skip gram model, softmax is computationally very expensive, as
it requires scanning through the entire output embedding matrix (W_output) to compute the probability ...
- 139
0
votes
0
answers
8
views
Calculating noise distribution in skip gram negative sampling
I was referring to this article explaining skip-gram with negative sampling. It says we need to sample negative samples from noise distribution calculated as follows:
$$P_n(w) = \left(\frac{U(w)}{Z}\...
- 139
0
votes
0
answers
12
views
Why don't we use binary vectors in positional coding?
I found this article on positional encoding (https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3).
But I don't understand when the author says that you have to measure a ...
- 1
0
votes
0
answers
20
views
How to build a Resume-to-Job Description matcher based on a parsed JSON Resume dataset?
For my capstone project/internship I'm working on an "HR assistant" tool designed to help match, score and rank resumes given a job description and/or requirements. I have inquired about ...
- 1
0
votes
0
answers
7
views
Does word2vec skip gram involves softmax in the output layer
I was going through various pytorch and from-scratch implementations of skip-gram. I found following:
This implementaiton does not seem to use softmax
...
- 99
0
votes
0
answers
8
views
What models/techniques can I use to generalize industry specific datasets?
I have a few dictionaries pertaining to different industries (ie. tech, manufacturing, education, etc.). These dictionaries map phrases and keywords to a sentiment score. I'd like to create a ...
1
vote
1
answer
36
views
Topic classification on text data with no/few labels
I would like to achieve a classification of a text input into predefined categories.
From what I have understand unsupervised approach are unfeasible if my target label is something very rare in ...
- 45