Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
0
votes
1answer
4 views
1
vote
1answer
26 views

Can AI (NLP) convert user questions (text) into database SQL queries?

I have been reading about NLP but got confused and not able to figure out - if it is feasible for NLP to convert questions in natural language to transform into SQL queries (so that it can execute on ...
0
votes
1answer
14 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
0
votes
2answers
12 views

What does Conv1d do in a sentiment analysis?

I am doing some study on https://www.kaggle.com/anshulrai/cudnnlstm-implementation-93-7-accuracy I understand we need LSTM to capture the sequence of words in the sentience, but I am not quite ...
0
votes
1answer
22 views

What does the dimension represent in the GloVe pre-trained word vectors?

I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) to word embedding. I have a conceptual question: What is the difference between these files? On the other hand, what ...
3
votes
1answer
39 views

How to cluster n-grams?

I just wanted to know how to cluster n-grams based on their semantics. Like clustering together n-grams that are semantically similar by leveraging the distributional hypothesis suggesting that ...
1
vote
0answers
9 views

Training data requirements for NLP models

Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the ...
1
vote
1answer
26 views

How to choose solution - Neural Neworks or Scikit-Learn/Numpy/Pandas?

I am trying to solve a problem - categorising and routing service desk emails to concerned teams for resolution. Created and tested a model using Scikit-Learn, Numpy and Pandas. - Tokenized the email ...
0
votes
1answer
16 views

Custom POS tags with SpaCy for NER

Quite new to NLP and especially NER. I'm trying to train a NER model on a custom dataset. This is a dataset of houses for sale. As part of the entities I'm training the model to extract are reference ...
1
vote
1answer
10 views

What is word embedding and character embedding ? Why words are represented in vector with huge size?

In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and Why words are represented in vector and ...
0
votes
0answers
10 views

What's the actual model inside the blackbox of spacy predefined models

I currently use Spacy to do my nlp tasks and really like their pre-trained models (the one that can be loaded with nlp = spacy.load('en_core_web_sm') which enable ...
0
votes
0answers
26 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
0
votes
0answers
11 views

Is there a text counter part to mini-imagenet?

I've been in search of a text dataset similar to mini-imagenet (many classes with 100s of images for each class), where, instead of images for training examples, we have detailed physical descriptions....
0
votes
1answer
8 views

A Derivation in Combinatory Categorial Grammer

I am reading about CCG on page 23 of Speech and Language processing. There is a derivation as follows: (VP/PP)/NP , VP\((VP/PP)/NP) => VP? Can anyone example ...
2
votes
3answers
21 views

Are stopwords helpful when using tf-idf features for document classification?

I have documents of pure natural language text. Those documents are rather short; e.g. 20 - 200 words. I want to classify them. A typical representation is a bag of words (BoW). The drawback of BoW ...
0
votes
0answers
6 views

Combining decision trees and neural networks for classifying text with metadata . How to combine and train?

I have a multi-label classification problem where the input consist of free text, with metadata such as categories (from a fixed, limited set) associated with each text. The output consist of a set of ...
-1
votes
0answers
41 views

Predicting Distractor for QnA [closed]

Hello Everyone, I need a help for a NLP task . Problem Statement : The task is to build a model for QnA and predict it's most possible distractors. Given: Below is a sample of train dataset Train ...
0
votes
1answer
14 views

Rank links from rss feed

I am trying to create a script to filter the most "intersting" articles from an rss feed and rank them. ...
1
vote
1answer
17 views

How to find similar phrases

I have the following problem: I have created a customized Dictionary for getting used in some NLP tasks. I want to enhance my dictionary by finding phrases similar to the phrases I have in my ...
0
votes
0answers
9 views

nlp: Translation System: Transformer/GPT2 model: Why do we need to mask future tokens?

I am trying to understand the whole concept of masking the tokens in the transformer/gpt2 model. In this blog post, http://jalammar.github.io/illustrated-gpt2/ the author takes an example where " the ...
1
vote
1answer
1k views

Distractor Generation for Multiple Choice Questions

I'm currently working on generating distractor for multiple choice questions. Training set consists of question, answer and 3 distractor and I need to predict 3 distractor for test set. I have gone ...
1
vote
0answers
11 views

Obtain Document-Topic matrix from NMF

I recently used NMF model provided by SkLearn to obtain Topics and terms under it. SKlearn provided the following code in their documentation: ''' Get Words in the topics def print_top_words(model, ...
1
vote
0answers
20 views

Embedding representation for a document?

Is averaging sentence embeddings, the right way to get representation for documents. Say I have a list of sentence embeddings representing symptoms. A data point looks like these: x|S1,S2,S3 --> Y|D1,...
0
votes
0answers
16 views

Intuition for inference of doc2vec models, on document parts

I am trying to understand how doc2vec models perform during inference on documents when we split them in various ways. Example document: ...
2
votes
1answer
32 views

What is auxiliary loss in Character-level Transformer model?

I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and ...
1
vote
1answer
22 views

NER with Unsupervised Learning?

If we treated NER as a classification/prediction problem, how would we handle name entities that weren't in training corpus? For example, "James was born in England." James was labeled as a PERSON ...
2
votes
2answers
38 views

How to extract assignment from natural language text?

I'm a bit new to NLP/IE. I'm looking for the task within NLP/IE that would be concerned with extracting a value that has been assigned. For instance, given the text "The value is 45.1hz" or "The ...
0
votes
0answers
8 views

Probability Mass Function of Trigrams of DevSet in Linear Interpolation

I am reading the text on page 12-13, the A linearly interpolated trigram model is derived is defined in terms of the trigram, bigram, and unigram maximum-likelihood estimates $q(w|u,v)=λ_1 ×q_{ML}(w|...
1
vote
0answers
16 views

N-Gram Linear Smoothing

In slide 61 of the NLP text, to smooth out the n-gram probabilities, we need to find the lambdas the miximazies a probability to held-out set given in terms of M(λ1, λ2, ...λ_k). What does this ...
0
votes
1answer
12 views

Best practice on count of manual annotations for building criminal detection from news articles?

We have 7 million news articles corpus, which we want to classify into crimes or non-crimes and further identify criminals by using NERs/annotating criminals, crime manually. For creating a model that ...
1
vote
3answers
37 views

Using Google Translate API to create a Translation Dataset

Is it a good idea? ;-) Is it legal to do so? Is it legal to release such a dataset to public? Say I have a language X for which I want to create a dataset for translation to/from English, for which I ...
0
votes
1answer
19 views

Analyzing Sentiments of Financial News related to a Company

I'm trying to build a model which gives me the sentiments of the Financial News related to a company and I want to predict the stock price accordingly. But the major problem that I'm facing is ...
1
vote
1answer
34 views

how to use word embedding to do document classification etc?

I just start learning NLP technology, such as GPT, Bert, XLnet, word2vec, Glove etc. I try my best to read papers and check source code. But I still cannot understand very well. When we use word2vec ...
0
votes
0answers
14 views

How to identify occupations in the OSHA data file?

My dataset is from the injury and illness recording, osha, dataset which looks something like this: So the second column contains the title for ...
0
votes
1answer
30 views

How to validate a clustering model without a ground truth?

Im dealing with a dataset (text messages about source code comments) that are not labeled. I don't have a assumption about the implicits classes in this dataset. I want to discovery (by clustering) ...
0
votes
2answers
47 views

Using LSTM for binary text Classification, getting almost same accuracy at each epoch

I am doing Twitter sentiment classification. For that I am using LSTM with pretrained 50d GloVe word embeddings(not training them as of now, might do in future). The tweets are of variable lengths ...
1
vote
0answers
28 views

Sentiment Analysis: using a dataset (IMDB reviews) to train a neural-net and using it to predict entirely different datasets (Political articles)

We need to analyse a lot of articles relevant to political instability in a given country (things like the possibility of a coalition / a snap election etc). The problem is that I could not find any ...
1
vote
2answers
44 views

How can I get probabilities of next word with ELMO?

ELMO is a language model, build to to compute the probability of a word, given some prior history of words seen. How can I get this probability from pretained ELMO model?
0
votes
1answer
17 views

Is it possible to create a rule-based algorithm to compute the relevance score of question-answer pair?

In information retrieval or question answering system, we use TD-IDF or BM25 to compute the similarity score of question-question pair as the baseline or coarse ranking for deep learning. In ...
2
votes
1answer
24 views

Why result of CountVectorizer * TfidfVectorizer.idf_ is different from TfidfVectorizer.fit_transform()?

I have dataframe: df = pd.DataFrame({'docs': ['gamma alfa beta beta epsilon', 'beta gamma eta',], 'labels': ['alfa alfa beta', 'gamma fi']}) I do count ...
0
votes
0answers
6 views

Doubt on formulating cost function for GloVe

I'm reading the notes here and have a doubt on page 2 ("Least squares objective" section). The probability of a word $j$ occurring in the context of word $i$ is $$Q_{ij}=\frac{\exp(u_j^Tv_i)}{\sum_{w=...
-1
votes
0answers
30 views

How to prepare data

I have 3 tables: Product table: product ID ( numerical) product ingredients ( words each in one column: ingredient1, ingredient2, etc ) product brand( words) product url product name Customer ...
2
votes
1answer
48 views

How to justify the usage of 200 dimensions in word vectors instead of the 300 dimensions?

When employing machine learning methods in NLP, most of studies use 200 or 300 dimensional vectors. 300 dimensional embeddings carry more information and this, therefore, is considered to produce ...
1
vote
1answer
33 views

Generate new sentences based on keywords

For example, for a domain specific neural network in Fashion, with the Keywords light, dress, orange, cotton. It could output: This gorgeous orange summer dress is great for wearing on sunny camping ...
0
votes
1answer
37 views

NMT, What if we do not pass input for decoder?

For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output. What ...
1
vote
1answer
17 views

Infer family type, size from reviews

I have a bunch of reviews: ...
2
votes
1answer
22 views

Google NLP AutoML

I am doing research for Google NLP AutoML, What methodologies they have used, techniques, models, feature selection, hyper parameter optimization, etc. I could not find any paper on how google built ...
2
votes
0answers
18 views

NLP based Data Preprocessing Method to Improve Disease Name Prediction Using CRF and Word Embedding

I built a model( using CRF along bi lstm) to Predict New Disease Name/Entities from medical text data but the problem is Disease name appears only 5,6 times in 1 text file but on average text file ...
2
votes
1answer
23 views

Spacy word embeddings for sentence

Spacy offers pre-trained vectors for words. However I have notices that you can get vectors for sentences too: spacy_nlp('hello I').has_vector == True However I ...
0
votes
0answers
8 views

Sparse Matrix TextRank function?

I want to run the standard TextRank algorithm on a quite large corpus (100,000 documents x 15+- sentences per document). My purpose is text summarization. The standard implementation I am seeing ...