We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

Filter by
Sorted by
Tagged with
2
votes
1answer
21 views

Is there a way to cluster words based on how similarly they sound?

I have a list of words for a fictional world I've made (don't judge lol). My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build ...
0
votes
0answers
32 views

Can Amazon Comprehend (Medical) be used for autocompletion of words or phrases?

"Amazon Comprehend Medical can accurately identify abbreviations, misspellings, and typos in medical text" source here. As such I believe one could use it to develop an autocorrect service, that ...
0
votes
1answer
25 views

Is there any text similarity databse available for phrases?

I want to train my application for phrase similarity. I want my model to predict similarity score for phrases as shown in below examples. ex- ...
1
vote
0answers
28 views

How is maximizing L(lambda1, lamda2, lamda3) equivalent to minimizing perplexity?

In language modeling, L(lambda1, lambda2, lambda3) is defined as: Sum(count of trigram(u,v,w) x q(w|u,v)) where u, v, w are words in the corpus and perplexity ...
2
votes
1answer
264 views

Trying to implement a “smart compose” feature

I found this post on Gmail's smart compose feature, and it got me thinking about trying to implement it myself. https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html The text ...
1
vote
0answers
59 views

How to extract name of objects from technical description (NLP)

I have a list of technical descriptions of mechanical parts such as Relay Compressor A4u8skk Relay-Start-Compressor sdfvsh2 Relay-Start-Compressor-Copelan 05-569-21 Compressor-J4KS0AK-5gss-fj2 ...
0
votes
1answer
26 views

Identify specify areas in the text

I'd be interested in identifying various areas in the text message. Let's say I have a text containing some introduction, then there is a poem and at the end there are some urls to some web pages. I'...
1
vote
0answers
320 views

Word classification (not text classification) using NLP [closed]

I have been trying to extract Person name and Company name out of string. But, I have been facing lot of difficulties. I have a dataset of names and a dataset of company names. In the string, I wish ...
0
votes
0answers
9 views

Replacement for English Wizard

I am using the English Wizard to access the SQL database. Is there any other 3rd party tool similar to the English Wizard that will work on web and desktop application?
1
vote
0answers
49 views

Job Recommendation System

I am building a Job Recommendation System where I have Student Data for different subjects in Machine Learning(Data Viz, Python, Statistics, etc) and their skills from the resume. Need to Recommend ...
2
votes
3answers
116 views

Which neural network to choose for classification from text/speech?

I am considering two tasks: Dialog Act Classification from Text (e.g. classify to: question; opinion; ...) Emotion Recognition from Speech (e.g. happy; calm; sad; ...) Which DL model should perform ...
1
vote
2answers
43 views

Continous bag of words claimed to be unsupervised, how is it working?

I'm following these two lectures on CBOW and skip-gram word2vec models. The first is lec 12 and the next lec 13 of a deep learning series https://www.youtube.com/watch?v=syWB-YMYZvI https://www....
1
vote
1answer
458 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
1
vote
0answers
24 views

Is there a way to automatically generate a string given an input of another string? [closed]

I am fairly new to data science in general, and I'm wondering if it is possible to use machine learning/natural language processing to generate definitions for various terms. My team has recently ...
2
votes
3answers
116 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
1
vote
0answers
30 views

How to learn word embedding from a context on the fly?

Consider the fictional word tahiliuk in the sentence “We found a small, fluffy tahiliuk running around our garden.” While hearing a new word used in context, people are remarkably adept at inferring a ...
0
votes
1answer
25 views

Find correct forms of a word

I'm trying to find all grammatical forms for words from a dataset. I'm new to NLP. I'm using a combo: spaCy (to get a base form of a word), ...
0
votes
2answers
131 views

How do you measure performance for word prediction tasks?

Say I have to predict the next word in a sentence, given the initial few words. Suppose the prefix is "I went to _____". This prefix is common enough that it might appear 10 times in the training ...
1
vote
0answers
22 views

TextRank algorithm for Web content

I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags ...
0
votes
1answer
20 views

I am getting a Type Error in this Line

Diff = [i - j for i,j in zip(text_features, author_signature)] Diff is a List , text_features = [1, 2, 3] , author_signature = [3, 2, 1]
1
vote
0answers
191 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
4
votes
4answers
388 views

Classification of Conversations in Text

I am trying to pick a technique for classifying conversational text. I am concerned about treating the problem at a level of fidelity of each individual message because people often say things like, "...
2
votes
1answer
28 views

Confidence Intervals for Multi-Categorical Votes

I have an ngram-based language model that produces a long tag list for a given sentence. For example, the just-previous sentence, broken into bigrams, and run through the model might produce something ...
1
vote
1answer
226 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
1
vote
0answers
37 views

Using an ontology to recognize named entities in free text

I'm trying to solve a fairly basic problem in NPL efficiently. What tool or software package would you use to identify the words, or group of words that are part of an given ontology within a free ...
1
vote
0answers
19 views

What approach to use to detect violations of media ethics in news?

I've been trying to come up with a solution to detect violations of media ethics in news articles. For example, I need to detect if an article has mentioned the name of a victim of rape. So far, I've ...
1
vote
0answers
28 views

Common deep learning practices in NLP for text classification

Are there any articles about best modern techniques in text classification (not only for English texts, but in general)? In particular, I'm interested: what kind of text preprocessing techniques are ...
2
votes
1answer
162 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...
1
vote
0answers
31 views

Check If Answer is Correct by Similarity

I am new to data science and machine learning. Let's say that I have a question, and some correct answers for that question (for example, 10 correct answers). Is there a way to get a new answer as ...
1
vote
2answers
2k views

Difference between IOB and IOB2 format?

I have to tag a dataset for NER. I came across conll2002/esp. What I understand so far, in IOB2 format if I want to tag '...
2
votes
0answers
24 views

How do I perform Sentiment Analysis on Tweets in the following pattern:

I have tweets obtained based on matches (football) before the match begins. I have tweets which specify a team will win 3-1 and so on which are easily analyzed using regular expressions. I am facing ...
0
votes
1answer
56 views

Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic?

I need to classify whether a given review or comment is a complaint or appreciation. This is planned to be used in multiple places, product review pages of own site as well as facebook and twitter. ...
1
vote
0answers
11 views

Grouping prefix and term queries

Consider the following context: You have a website that is offering a search function, which produces a list of matching articles. The search has prefix functionality: if you are looking for "...
1
vote
1answer
34 views

how to encode labels for relationship extraction

I am trying to extract relationship from text. So, lets take the following text "He went to movie. but, they went to school" So, here the relationship's are "He and Movie", "they and school". How ...
1
vote
0answers
157 views

Loss function for Hierarchical Multi-label classification

I am looking to try different loss functions for a hierarchical multi-label classification problem. So far, I have been training different models or submodels (e.g., a simple MLP branch inside a ...
2
votes
0answers
94 views

Joint embedding of word and image

I often see some papers, in which the authors do the point-wise multiplication of word embedding and the image embedding. As the image shows below, my question is how come the implementation works?? I ...
3
votes
0answers
87 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
1
vote
0answers
78 views

How are natural language generation algorithms given a target

I've started learning about NLP and NLG and I'm fascinated! I've been blown away by the things I've seen from NLP; but I have a few questions about NLG. All my questions boil down to this: Given a ...
1
vote
1answer
73 views

Question classification

I have 10 classes and 10-15 questions in each class . Given a new question, I want to find the class to which the question is most similar?
1
vote
1answer
39 views

How to find impactful words affecting classification?

So I know there are many methods to classify sentences into types. Like in sentiment analysis (positive, negative, neutral), spam emails (spam, not spam), etc. The thing I want to ask is how would I ...
0
votes
1answer
29 views

Tools and Techniques for Analyzing German Automotive Discussion Forum Posts [closed]

I work for a German online disussion forum around all things automotive, a bit like a “StackOverflow for cars”, if you will. We would like to train a model using TensorFlow with our high quality ...
1
vote
1answer
255 views

Data scraping & NLP?

I'm scraping data from Bing search results for (non-commercial purposes, of course) on Python using BeautifulSoup. I've entered an Indian dessert name, called 'rasmalai' as the word that I am focusing ...
1
vote
0answers
27 views

Predicting a new document [closed]

I have a document, (purchase agreement) of approx. 100 pages. This document is sent from buyer to seller several times, and each time there is a negotiation. Negotiation could be anything. For eg. ...
5
votes
1answer
467 views

Is it valid to include your validation data in your vocabulary for NLP?

At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, ...
1
vote
1answer
367 views

transfer learning with sentiment analysis?

The question is how good and what are some things to keep in mind when sentiment analysis models are tested on different datasets than they are trained on. Say the task is to perform sentiment ...
1
vote
2answers
51 views

How should I format input and output for text generation with LSTMs

I'm attempting to generate a response to an input line of text using an LSTM. I've considered various forms of input, including one-hot encoding each character in the line and passing each input line ...
2
votes
1answer
320 views

Fine-tuning NLP models

In computer vision, if we don't have a large training set, a common method is to start with a pre-trained model for some related task (e.g., ImageNet) and fine-tune that model to solve our problem. ...
1
vote
0answers
18 views

Machine Learning Text Categorization with Data in multiple Languages

I want to label natural text Dokuments in various different categories (arround 400 different categories) using Neural Networks. I have around 50.000 documents already labeled. The problem is around ...
11
votes
3answers
16k views

Natural Language to SQL query

I have been working on developing a system "Converting Natural Language to SQL Query". I have read the answers from the similar questions, but was not able to get the information that I was looking ...
1
vote
1answer
299 views

How to combine sparse text features with user smile for sentiment classification? [closed]

I am trying to perform sentiment classification task where I have some text and some information about whether the user smiled or not. Now when I use count-vectorizer to convert my text to feature ...