Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

0
votes
0answers
15 views

Text classification problem [on hold]

Am asked to extract tenders titles in a specific work-field (expl: oil sector) from the web , am really beginner in NLP , what steps and methods do i have to use ?
2
votes
1answer
15 views

Is there a way to cluster words based on how similarly they sound?

I have a list of words for a fictional world I've made (don't judge lol). My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build ...
0
votes
0answers
9 views

Can Amazon Comprehend (Medical) be used for autocompletion of words or phrases?

"Amazon Comprehend Medical can accurately identify abbreviations, misspellings, and typos in medical text" (source: https://aws.amazon.com/comprehend/medical/). As such I believe one could use it to ...
0
votes
1answer
15 views

Is there any text similarity databse available for phrases?

I want to train my application for phrase similarity. I want my model to predict similarity score for phrases as shown in below examples. ex- ...
1
vote
0answers
21 views

How is maximizing L(lambda1, lamda2, lamda3) equivalent to minimizing perplexity?

In language modeling, L(lambda1, lambda2, lambda3) is defined as: Sum(count of trigram(u,v,w) x q(w|u,v)) where u, v, w are words in the corpus and perplexity ...
0
votes
0answers
10 views

Trying to implement a “smart compose” feature

I found this post on Gmail's smart compose feature, and it got me thinking about trying to implement it myself. https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html The text ...
1
vote
0answers
30 views

How to extract name of objects from technical description (NLP)

I have a list of technical descriptions of mechanical parts such as Relay Compressor A4u8skk Relay-Start-Compressor sdfvsh2 Relay-Start-Compressor-Copelan 05-569-21 Compressor-J4KS0AK-5gss-fj2 ...
0
votes
0answers
17 views

Identify specify areas in the text

I'd be interested in identifying various areas in the text message. Let's say I have a text containing some introduction, then there is a poem and at the end there are some urls to some web pages. I'...
1
vote
0answers
44 views

Word classification (not text classification) using NLP [closed]

I have been trying to extract Person name and Company name out of string. But, I have been facing lot of difficulties. I have a dataset of names and a dataset of company names. In the string, I wish ...
0
votes
0answers
24 views

How to check whether word is abbreviated or not using python [migrated]

How to check whether word is abbreviated or not in dataframe column using python. For instance, U.S.A value I need to detect it as abbreviation. Is there any dictionary present for this?
0
votes
0answers
8 views

Replacement for English Wizard

I am using the English Wizard to access the SQL database. Is there any other 3rd party tool similar to the English Wizard that will work on web and desktop application?
1
vote
0answers
21 views

Job Recommendation System

I am building a Job Recommendation System where I have Student Data for different subjects in Machine Learning(Data Viz, Python, Statistics, etc) and their skills from the resume. Need to Recommend ...
0
votes
2answers
71 views

Which neural network to choose for classification from text/speech?

I am considering two tasks: Dialog Act Classification from Text (e.g. classify to: question; opinion; ...) Emotion Recognition from Speech (e.g. happy; calm; sad; ...) Which DL model should perform ...
0
votes
0answers
10 views

NLP - Researches about data oriented text generation

I am relatively new to NLP. I am looking for a guidance as to where I can look to find about current researches in data driven text-generation, or data driven chatbot. Has there been some researches ...
0
votes
0answers
19 views

What is the beam density in beam search algorithm witha application to spacy ner model?

I am retraining spacy's NER model using the method described in this link I am using also the method described here in order to get the probabilities of the results. The beam search algorithm seems ...
0
votes
0answers
11 views

Evaluating Performance of Unsupervised Learning Algorithm

I'm trying to match two Documents using Doc2Vec Algorithm And so far it is giving satisfactory results, However I'm unable to know if my model is improving or not. I'm now stuck in the thought process ...
1
vote
2answers
25 views

Continous bag of words claimed to be unsupervised, how is it working?

I'm following these two lectures on CBOW and skip-gram word2vec models. The first is lec 12 and the next lec 13 of a deep learning series https://www.youtube.com/watch?v=syWB-YMYZvI https://www....
0
votes
0answers
7 views

Hierachical Softmax Example In Deep Learning Book

I was learning neural network using the book "Deep Learning" by Ian Goodfellow, Yoshua Bengio and Aaron Courville. Section 12.4.3.2 has an example to show the computation cost improvement from ...
1
vote
1answer
105 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
0
votes
0answers
25 views

How CNN contributes differently in sentiment classification task than RNN?

I would like to know how fundamentally CNN is different from RNN (Many to one) for sentiment classification task. More specifically, what CNN models can learn from data that the RNN can't learn or ...
1
vote
0answers
19 views

Is there a way to automatically generate a string given an input of another string? [closed]

I am fairly new to data science in general, and I'm wondering if it is possible to use machine learning/natural language processing to generate definitions for various terms. My team has recently ...
1
vote
2answers
45 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
1
vote
0answers
24 views

How to learn word embedding from a context on the fly?

Consider the fictional word tahiliuk in the sentence “We found a small, fluffy tahiliuk running around our garden.” While hearing a new word used in context, people are remarkably adept at inferring a ...
0
votes
1answer
21 views

Find correct forms of a word

I'm trying to find all grammatical forms for words from a dataset. I'm new to NLP. I'm using a combo: spaCy (to get a base form of a word), ...
0
votes
1answer
60 views

How do you measure performance for word prediction tasks?

Say I have to predict the next word in a sentence, given the initial few words. Suppose the prefix is "I went to _____". This prefix is common enough that it might appear 10 times in the training ...
1
vote
0answers
14 views

TextRank algorithm for Web content

I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags ...
0
votes
1answer
18 views

I am getting a Type Error in this Line

Diff = [i - j for i,j in zip(text_features, author_signature)] Diff is a List , text_features = [1, 2, 3] , author_signature = [3, 2, 1]
1
vote
0answers
40 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
3
votes
2answers
81 views

Classification of Conversations in Text

I am trying to pick a technique for classifying conversational text. I am concerned about treating the problem at a level of fidelity of each individual message because people often say things like, "...
2
votes
1answer
26 views

Confidence Intervals for Multi-Categorical Votes

I have an ngram-based language model that produces a long tag list for a given sentence. For example, the just-previous sentence, broken into bigrams, and run through the model might produce something ...
1
vote
1answer
49 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
1
vote
0answers
23 views

Using an ontology to recognize named entities in free text

I'm trying to solve a fairly basic problem in NPL efficiently. What tool or software package would you use to identify the words, or group of words that are part of an given ontology within a free ...
1
vote
0answers
19 views

What approach to use to detect violations of media ethics in news?

I've been trying to come up with a solution to detect violations of media ethics in news articles. For example, I need to detect if an article has mentioned the name of a victim of rape. So far, I've ...
0
votes
0answers
49 views

Recommending top answerers for a Question on Quora/Stackexchange sites

I want to make a tool that will tell me who can potentially answer a question on Quora or StackExchange. The tool will take input the text of a question, and output a sorted list of users who can ...
1
vote
0answers
22 views

Common deep learning practices in NLP for text classification

Are there any articles about best modern techniques in text classification (not only for English texts, but in general)? In particular, I'm interested: what kind of text preprocessing techniques are ...
2
votes
1answer
80 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...
1
vote
0answers
18 views

Check If Answer is Correct by Similarity

I am new to data science and machine learning. Let's say that I have a question, and some correct answers for that question (for example, 10 correct answers). Is there a way to get a new answer as ...
1
vote
1answer
462 views

Difference between IOB and IOB2 format?

I have to tag a dataset for NER. I came across conll2002/esp. What I understand so far, in IOB2 format if I want to tag '...
2
votes
0answers
21 views

How do I perform Sentiment Analysis on Tweets in the following pattern:

I have tweets obtained based on matches (football) before the match begins. I have tweets which specify a team will win 3-1 and so on which are easily analyzed using regular expressions. I am facing ...
0
votes
1answer
37 views

Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic?

I need to classify whether a given review or comment is a complaint or appreciation. This is planned to be used in multiple places, product review pages of own site as well as facebook and twitter. ...
1
vote
0answers
10 views

Grouping prefix and term queries

Consider the following context: You have a website that is offering a search function, which produces a list of matching articles. The search has prefix functionality: if you are looking for "...
1
vote
1answer
23 views

how to encode labels for relationship extraction

I am trying to extract relationship from text. So, lets take the following text "He went to movie. but, they went to school" So, here the relationship's are "He and Movie", "they and school". How ...
0
votes
0answers
31 views

NLP research: Emotive conjugation

Is there any formal NLP research into emotive conjugation (aka Russel's conjugation)? Here are some examples of emotive conjugation by Bertrand Russell that are shown on the Wikipedia page: I am ...
0
votes
0answers
14 views
1
vote
0answers
101 views

Loss function for Hierarchical Multi-label classification

I am looking to try different loss functions for a hierarchical multi-label classification problem. So far, I have been training different models or submodels (e.g., a simple MLP branch inside a ...
0
votes
0answers
21 views

Unsupervised answering for a predefined set of questions

I am working on a project to read up a text segment and find answers to a specific set of questions, in order to do some information extraction. I have a set of text corpus (each of about 3000 words),...
0
votes
0answers
16 views

How to extract event from news article and merge similar event together

TL;DR Is there a way to extract events from news and cluster similar news event together? I am following the ACE methodology currently, is that a viable way? Longer version: I am currently working ...
2
votes
0answers
30 views

Joint embedding of word and image

I often see some papers, in which the authors do the point-wise multiplication of word embedding and the image embedding. As the image shows below, my question is how come the implementation works?? I ...
3
votes
0answers
57 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
1
vote
0answers
27 views

How are natural language generation algorithms given a target

I've started learning about NLP and NLG and I'm fascinated! I've been blown away by the things I've seen from NLP; but I have a few questions about NLG. All my questions boil down to this: Given a ...