Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural ...

1
vote
1answer
6 views

Is it possible to use word2Vec to derive hyperonymy (hyponymy or ISA relation)?

It's easy to have hyperonymy in WordNet, e.g. to know that "tea" is a case of "beverage". Is it possible to use word2Vec in this way?
0
votes
0answers
6 views

Are the Doc2vec paper and framework explanation the same?

I am looking into doc2vec since it shows promising results in several articles. When I read the paper by Le & Mikolov, I was under the assumption that the paragraph vector was also used to predict ...
0
votes
0answers
14 views

Huge difference in accuracy when using two datasets (instead of one) for text classification in Python

Recently I started reading more about NLP in order to learn more about the subject. The problem that I've encountered, now that I'm trying to make my own classification algorithm (the text sends a ...
1
vote
2answers
23 views

How to create clusters based on sentence similarity?

I have data which looks like following. Data is a group of sentences which are similar, but have few unique words in between like TABLEA, TABLEB etc. ...
0
votes
1answer
9 views

How to improve accuracy of Named entity recognition (NER) tagger on local data?

I am using NER from spacy. Its giving incorrect results for few words. Its trained on general dataset. How can I customize on my local data. For example, ...
0
votes
0answers
8 views

Memory Neural Network to convert binary vector to string

I am using a mapping function that transforms each word into a binary vector, for example: ...
0
votes
1answer
37 views

Possible reasons for word2vec learning context words as most similar rather than words in similar contexts

I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave ...
0
votes
1answer
18 views

Document similarity matching between Doc2Vec documents

I am creating a Doc2Vec model out of hundreds of PDF documents. I have 17 documents that are part of this Doc2Vec that I want to use to check similarity with other documents in the Doc2Vec model. ...
1
vote
0answers
12 views

How can I select a similarity threshold value for strings?

I have several large data sets of names (i.e., individuals, groups, and companies) where there is a lot of similar, but not quite the same data. For example, a corporation might be listed as "Acme ...
0
votes
0answers
11 views

How to segregate a language's topics into complexity-level buckets?

Problem Statement: Divide a Programming Language (say JS) into topics. Then, segregate the topics into three buckets based on the complexity of the concept: Beginner, Intermediate and Advanced. ...
2
votes
1answer
15 views

Why ELMo's word embedding can represent the word better than glove?

I have read the code of ELMo: https://github.com/allenai/bilm-tf Based on my understanding, ELMo first init an word embedding matrix A for all the word and then ...
1
vote
1answer
35 views

What are common ways to identify subject and object from a question?

I am looking for a way to extract the potential subject and object from a question in French. For the moment, I am building some handmade rules. Alternatively, I started to think of using an already ...
1
vote
0answers
26 views

Skip-thought models applied to phrases instead of sentences

My goal is to build a statistical model with domain specific phrase embeddings. To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence ...
0
votes
0answers
11 views

Doc2Vec Multiple Label Vectors

I have been exploring gensim's Doc2Vec library and it produces some pretty interesting results, and I'm beginning to explore multi-label embeddings. Through Radim's tutorial I understood that the ...
0
votes
0answers
7 views

Finding ideas in google corpus [closed]

There is the google corpus of all books google digitized. I want to write down an idea I have about something. Maybe 200 words. Then I want this encoded and compared to the google corpus. The desired ...
0
votes
0answers
10 views

Sequence tokenization and pretrained embedding layers

Sequence tokenization and pretrained embedding initialization - say you have a unique (but not huge) corpus of texts, and you also load a pretrained embedding vector (for example GloVe-100d). What's ...
0
votes
2answers
24 views

Is this a good approach to classify tickets which contains description and logs?

I want to classify a dataset of support tickets which mostly contain text in the description field and sometimes server logs in a separate field. The log field is not always there but when it's ...
0
votes
1answer
21 views

Handle 50,000 classes in OneVsRestClassifier

I'm new to data science and NLP. I'm trying to solve a problem that is having 1 million rows and some 50,000 distinct classes. The dataset has some text column as a predictor and the other one is the ...
0
votes
0answers
18 views

I fail to build a decent NLP neural network (sequence generation)… Any advice?

So, my objective is to build a decent (and relatively simple) neural network able to predict the next word of a sentence, given the previous words. So far, the system i have consists in: A Word2vec ...
0
votes
0answers
13 views

generating text with neural network, how to judge against perplexity scores

To illustrate my problem, I'm using fastai's AWD-LSTM implementation. But the question is about judging perplexity scores in general. The code is included below, but only so you can reproduce my ...
1
vote
0answers
7 views

What does “False Alarm per Hour” mean in charts in keyword spotting papers/ articles?

I'm looking at papers and articles and they always have a similar chart like the below from (https://medium.com/@alirezakenarsarianhari/yet-another-wake-word-detection-engine-a2486d36d8d4) I get ...
6
votes
1answer
136 views

On a multi lingual sentiment corpus

I am looking to compile a sentiment corpus for news articles in multiple languages (~100k per lang. for a machine learning experiment) where each article is labeled positive, neutral, or negative. I ...
0
votes
0answers
17 views

What is the best way to use word2vec for bilingual text similarity?

I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this: ...
1
vote
0answers
16 views

What are common data augmentation techniques for nlp in general and for chatbot use specifically (in rasa)?

I am trying to build a chatbot using RASA The process to build the chatbot can summarized in two step: First identify the intent and the entities in the user's question Second, based on a history of ...
1
vote
0answers
31 views

Combine multiple features for text classification

Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. I'm trying to make my own classification algorithm (the text sends a positive/...
4
votes
0answers
42 views

any efficient way to find surrounding adjective/verbs with respect to the target phrase in python [updated]?

I am doing sentiment analysis on given documents. My goal is to find out the closest or surrounding adjective words with respect to the target phrase in my sentences. I do have an idea how to extract ...
1
vote
0answers
11 views

How to measure word similarity using wordnet for the information theoretic definition as detailed in Resnik 1995? [closed]

Resnik 1995 equation 3 uses count(n) to define P(c). What is count(n)? Any solved example would be appreciated.
1
vote
1answer
13 views

Grouping domain specific words/phrases with same meaning

I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/...
0
votes
0answers
13 views

NLP - Gensim lemmatize not lemmatizing a particular bigram

In the following code I don't understand why the bigram customer_service is coming up as empty. All other bigrams seem to work fine. ...
0
votes
1answer
14 views

Composing phrases into a grammatically correct sentence?

I'm wondering if there exist any models which could take in an ordered list of phrases without punctuation and generate a grammatically correct sentence from it. For example, for the input: ["My dog"...
2
votes
1answer
50 views

NLP algorithms for categorizing a list of words with specific topics

Currently I am using LDA to apply topic modeling to a corpus. Since LDA is unsupervised, it returns a set of words for a given 'topic' but doesn't necessarily specify the topic itself. I was wondering ...
0
votes
1answer
18 views

When one model is superior in real world use?

I have an NLP neural network that I have developed with Keras for multi-label classification. I have fit the model several times and save the best results (via best validation accuracy score) after ...
0
votes
0answers
19 views

Classification Text with fragmentation Dataset to analyze class by class

I am working with classifying texts and each class in my classification has 'similar' characteristics, that is, standards that classify it as that characteristic. I separated my Dataframe from these ...
0
votes
1answer
20 views

Find correct forms of a word

I'm trying to find all grammatical forms for words from a dataset. I'm new to NLP. I'm using a combo: spaCy (to get a base form of a word), ...
0
votes
0answers
17 views

Multiple entity extraction with character level RNN

I'm training a neural network to extract a certain kind of entities in a sentence (e.g. company names in a news title). Since I'm handling a multi-language corpus (especially CJK), which could be very ...
0
votes
0answers
22 views

Is this a known measure wrt to confusion matrix

We're working on a clinical NLP problem and have devised a method for combining output from multiple annotator systems into a merged annotation system. To do this, we are comparing each annotation ...
0
votes
1answer
57 views

How to use two different datasets as train and test sets?

Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. The problem that I've encountered, now that I'm trying to make my own ...
2
votes
1answer
61 views

TF-IDF Features vs Embedding Layer

Have you guys tried to compare the performance of TF-IDF features* with a shallow neural network classifier vs a deep neural network models like an RNN that has an embedding layer with word embedding ...
1
vote
1answer
49 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
0
votes
1answer
22 views

How do you measure performance for word prediction tasks?

Say I have to predict the next word in a sentence, given the initial few words. Suppose the prefix is "I went to _____". This prefix is common enough that it might appear 10 times in the training ...
0
votes
0answers
23 views

How do I gather negative samples for CBOW in word2vec?

I am trying to write the cbow part of wor2vec implementation, and I am not quite sure what would be qualify as a an appropriate negative sample needed for training. Lets say we have ...
1
vote
0answers
10 views

TextRank algorithm for Web content

I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags ...
1
vote
1answer
26 views

How to extract the positions of employee from raw text

I have raw text like "Mr John Fullerton is Chief Executive Officer and Managing Director of Australian Rail Track Corporation Ltd, and was appointed to the position in February 2011." I easily ...
0
votes
0answers
30 views

How to pass input to my text classifier

I have to classify meal times from natural language input Example data set in JSON format ...
1
vote
1answer
18 views

Identifying documents similar to specific clusters

Through performing clustering on a set of 1,000,000 text documents, I have identified 100 clusters. I am particularly interested in, say, 10 of the clusters. Imagine, I now have an additional set of ...
0
votes
1answer
36 views

Twitter tweet classification

I am trying to do a small project on my own to find out job openings using twitter data. I saved data using flume and converted it to .csv for analysis. My problem is i don't know how to classify ...
0
votes
1answer
15 views

practical improvements worth trying over plain LSTM in text classification?

I have a dataset of about 1 million tweets corresponding to about 30,000 user accounts, labelled with binary data (classifying the tweet as written by a bot). With that amount of data, I could use a ...
0
votes
1answer
32 views

Sequence models word2vec

I am working on data-set with more than 100,000 records. This is how the data looks like: ...
1
vote
0answers
17 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
0
votes
1answer
24 views

Opensource Speech Recognition Library that is secure and trained on large data [closed]

For all those who are working on developing a chatbot/assistant and care about the privacy of users consuming the speech recognition library, can you suggest an open souce library which is trained on ...