Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural ...

0
votes
1answer
5 views

practical improvements worth trying over plain LSTM in text classification?

I have a dataset of about 1 million tweets corresponding to about 30,000 user accounts, labelled with binary data (classifying the tweet as written by a bot). With that amount of data, I could use a ...
0
votes
0answers
12 views

Learning instructions from an instruction manual

I have a set of instruction manuals that follow a certain guideline and presents instructions about a specific product. What would be the best approach to use in order to create a machine learning ...
0
votes
0answers
26 views

Any practical improvements worth trying over plain LSTM in text classification?

I have a dataset of about 1 million tweets corresponding to about 30,000 user accounts, labelled with binary data (classifying the tweet as written by a bot). With that amount of data, I could use a ...
0
votes
1answer
18 views

Sequence models word2vec

I am working on data-set with more than 100,000 records. This is how the data looks like: ...
0
votes
0answers
5 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
0
votes
1answer
13 views

Opensource Speech Recognition Library that is secure and trained on large data [on hold]

For all those who are working on developing a chatbot/assistant and care about the privacy of users consuming the speech recognition library, can you suggest an open souce library which is trained on ...
0
votes
0answers
10 views

How to assign label to un-labeld text documents

I have a bunch of text documents and I would like to assign one or more than one label to each document. I know that autoencoders can be used in a semi-supervised setting to first cluster the ...
0
votes
1answer
32 views

Is my model over-fitting (LSTM,GRU)

I have small corpus max 150 text utterances, which is again distributed among 5 categories. To test I started with basic deep learning model, where i used word2vec embedding, added 1D convolution ...
0
votes
0answers
12 views

research oriented masters program in natural language processing [on hold]

I am interested in applying for a masters program with en emphasis on research in natural language processing/machine learning in US, but so far the only suitable program I've seen is MSLT in CMU. ...
0
votes
1answer
18 views

Alternatives to doc2vec?

What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the ...
1
vote
1answer
21 views

Stemmer/lemmatizer for Polish language

I'm looking for a stemmer/lemmatizer for Polish language, preferably in Python. What would you recommend? I have a list of ingredients in a recipe. Plural forms are inflected differently, depending ...
0
votes
0answers
18 views

Language Models vs Word Embeddings

I wanted to clear up my understanding of both Language Models and Word Embeddings and how they are related if at all. Language Models: There is the older counting model which consists of Markov ...
0
votes
0answers
13 views

Named entity co occurrences in sentences

I am researching about entity relations mined from articles. I am going to use the relations to form a social network that describes the relations between politicians in the news (strong relationships,...
0
votes
0answers
9 views

Text intent generation and request handling doubt

We are developing a chatbot for assistance in career related problems for high school students as a project for our Grad course For this purpose, we are going to make use of Deep Learning. The ...
0
votes
0answers
5 views

Public dataset for spanish news articles with title and content [migrated]

I am wondering if there are any public datasets of news in SPANISH similar to the one here: http://research.signalmedia.co/newsir16/signal-dataset.html Which consists mostly of ENGLISH news. There ...
2
votes
1answer
18 views

Classifying objects based of a varying number of the same type of feature vector for each object

For a congressional session, I have created a doc2vec model of speeches made. Using the vectors from this model, I have a dataset of each congressperson, their political affiliation, and a list of the ...
0
votes
0answers
11 views

I want to create a Data cleaning module for English language to clean certain Emails. How do i begin?

Here is an Example of an Email in my .xlsx file. End goal is to perform text summarization and Topic modelling of the various Email's present in the Excel file.
0
votes
0answers
16 views

What are the rules when extracting SVO triples from preprocessed text?

If you have some already preprocessed text that is tagged, what are the rules to extract SVO triplets if you want a triple like (word, word, word). Can you give the sentence as example and extract all ...
0
votes
1answer
16 views

How to fetch text from pdf to further proceed with question answer based model from the same document?

To illustrate the above title. Suppose you have a pdf document, which is basically scanned from hardcopy, now there are set of fixed questions to answer from the document itself. For an example a ...
2
votes
2answers
28 views

Spacy Returns Nonidentical Results for Doc. Examples?

I am getting started with Spacy, and I was just experimenting the given examples. It is weird though that when I run them over their containers using their experimental console it gives me different ...
1
vote
0answers
13 views

Model Joint Probability of N Words Appearing Together in a Sentence

Assume that we have a large corpus of texts to train with. Given N words as input, I want to model the joint probability $p(x_1, x_2, ..., x_N)$ of these words appearing together in a sentence. More ...
1
vote
1answer
16 views

Help with reusing glove word embedding pretrained model

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?
1
vote
1answer
24 views

Predicting topics for customer reviews based on topics mapped to n-grams?

I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic. Also I have a list of unigrams,bigrams and trigrams(not a part ...
2
votes
1answer
36 views

Find matching text from a text column

This is my first time to use Data Analytics tool to figure out a solution to a problem. I have a table with following columns ...
1
vote
0answers
18 views

Commercial Software for Interactive Machine Learning and Annotation in NLP

We are evaluating make or buy for a tool for interactive machine learning on legal text. With little labeling effort, it should enable for (ml layman) domain experts to build classifiers on a GUI. ...
1
vote
0answers
21 views

Can we use doc2vec to detect outlier documents?

I have a set of documents and I want to identify and remove the outlier documents. I am just wondering if doc2vec can be used for this task. Or are there any recently evolved, promising algorithms ...
0
votes
1answer
57 views

Increasing SpaCy max NLP limit

I'm getting this error: ...
1
vote
1answer
58 views

Interpretation of the loss function for word2vec

I am trying to understand the loss function which is used for the word2vec model, but I don't really follow the argumentation behind this video https://www.youtube.com/watch?v=ERibwqs9p38&t=5s, at ...
1
vote
0answers
15 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
0
votes
0answers
93 views

Extract variable names of formula from a document

How can I extract all of the variable names from a given research paper? For example Some applications of algebraic curves to computational vision For example, below is the text and I need to ...
1
vote
0answers
18 views

How to do good Keyword Extraction

I tried the sketch engine (no ad!) and I wonder what might be the underlying algorithms to do such a good keyword extraction. I have a document consitsing of rows of sentences and from each I want the ...
5
votes
1answer
89 views

Are there any good out-of-the-box language models for python?

I'm prototyping an application and I need a language model to compute perplexity on some generated sentences. Is there any trained language model in python I can readily use? Something simple like <...
0
votes
0answers
5 views

Parts-of-speech info for CMU Pronunciation Dictionary?

Is there a database of parts-of-speech for the words in the CMU Pronunciation dictionary? Wordnet has parts-of-speech, and I can do a simple join on the word between CMUdict and Wordnet, but this only ...
1
vote
3answers
82 views

Pass 2 different kinds of X training data to ML model simultaneously

I'm trying to classify if a book is fiction/nonfiction based on title and summary. This is 2 distinct types of information - is there a way to segment title and <...
1
vote
0answers
12 views

Efficient search for a Triples data [closed]

I have a text file which consists of ~10k triples in the format, Let say I need to extract all the relevant triples for the query "Tom and Jerry likes to play football", with this I am extracting NP(...
2
votes
1answer
19 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...
1
vote
1answer
26 views

Is there a dataset with news articles and their headlines? [closed]

I need a set of news headlines and articles to help me in a project on automatic summarization. Is there such a dataset or something similar?
3
votes
2answers
28 views

Check If Answer for a Question is Correct by Similarity

Let's say in an NLP problem, I have a question and some correct answers to that question (say, 10 correct answers). Is there a way to get a new answer as input, and "calculate" whether it is correct?...
2
votes
1answer
41 views

Are word embeddings further updated during training for document classification?

I am relatively new to the area of using word embeddings in NLP tasks. From a large corpus of documents, I train word2vec word embedding vectors and afterwards I am going to use these for document ...
1
vote
1answer
5 views

How to compute unseen bi-grams in a corpus (for Good-Turing Smoothing)

Consider a (somewhat nonsensical) sentence - "I see saw a see saw" The observed bi-grams would be: "I see""see saw""saw a"and,"a see". My aim is to smoothen out the probability mass of the bi-gram ...
1
vote
0answers
13 views

Check If Answer is Correct by Similarity

I am new to data science and machine learning. Let's say that I have a question, and some correct answers for that question (for example, 10 correct answers). Is there a way to get a new answer as ...
0
votes
0answers
32 views

Low accuracy issue - Multi lable classification on medical domain data

I am trying to build a model which helps in categorizing articles (pubmed data) which are of medical domain. An article can be associated with multiple categories. Overall there are 151 unique ...
1
vote
0answers
74 views

Difference between IOB and IOB2 format?

I have to tag a dataset for NER. I came across conll2002/esp. What I understand so far, in IOB2 format if I want to tag '...
2
votes
1answer
59 views

What to do if training loss decreases but validation loss does not decrease?

I am training a LSTM model to do question answering, i.e. so given an explanation/context and a question, it is supposed to predict the correct answer out of 4 options. My model architecture is as ...
1
vote
1answer
31 views

Extracting specific data from unstructured text - NER

I have a reasonably simple problem to solve. I need to extract reservations numbers from unstructured text. Based on my research, it seems to be an NER problem. Based on a visual analysis of the ...
0
votes
1answer
58 views

NER: Extracting entities from an article

Description : I have dataset of categorised articles and to extract specific values from respective categorised article I have regex created for each category. Aiming for: Nlp techniques which ...
0
votes
0answers
12 views

Long repetitive output after changing vocabulary in seq2seq model

I trained a neural question generation model, which produces sensible questions for the vocabulary that they distributed with the paper. I wanted to run the model on a different set of word embeddings ...
1
vote
1answer
15 views

Significance of log in DCG metric?

I am trying to make logical sense about why DCG is metric is formulated as such but I am not able to understand what's the need of taking the log of (1 + rank) term....
2
votes
2answers
37 views

In natural language processing, why each feature requires an extra dimension?

I am reading Machine Learning by Example. I am trying to understand natural language processing. The book used Scikit-learn's fetch_20newsgroups data as an example. The book mentioned that the text ...
1
vote
0answers
7 views

Reuters RCV2 topic classes hiearchy

I have the Reuters RCV2 multilingual text analysis data set. I am trying to figure out the hierarchy between the topic classes. For example, in RCV1 we know that C11, C12, C13 .. C15 all are the ...