Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

Filter by
Sorted by
Tagged with
0
votes
0answers
34 views

Can Amazon Comprehend (Medical) be used for autocompletion of words or phrases?

"Amazon Comprehend Medical can accurately identify abbreviations, misspellings, and typos in medical text" source here. As such I believe one could use it to develop an autocorrect service, that ...
1
vote
2answers
2k views

Difference between IOB and IOB2 format?

I have to tag a dataset for NER. I came across conll2002/esp. What I understand so far, in IOB2 format if I want to tag '...
0
votes
0answers
29 views

How can I find colours in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
1
vote
1answer
50 views

Why Heaps' Law Equation looks so different in this NLP course?

I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed ...
0
votes
1answer
27 views

How to cluster text-based software requirements

I'm beginner in deep learning and I'd like to cluster text-based software requirements by themes (words similarities/frequency of words) using neural networks. Is there any example/tutorial/github ...
1
vote
0answers
37 views

Implementation of NLP to categorize text into two categories

I can't discuss my actual dataset, so please bear with me. Let's say I have a dataset that contains a population of 20,000 examinations by a school principal. The principal is to record their ...
1
vote
0answers
184 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
0
votes
0answers
30 views

Given an input phrase, is there a way I can find the most similar phrase within a document?

I am completing a task where I need to retrieve the corresponding values to a set of given labels from many legal contracts. For example, one of the labels is "Floating rate payment dates" and it's ...
1
vote
0answers
19 views

Why is MLP working similar to RNN for text generation

I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus ...
1
vote
1answer
400 views

What are CRF (Conditional Random Field)

Looking for language modeling, I have been finding CRF in a lot of places which is but looking online for the same isn't actually helping me a lot. I referred Edwin Chen's blog and Ravish Chawala's ...
0
votes
1answer
48 views

How to calculate which word fits the best given a context and possible words?

I have this task for research purposes and searched a while for a framework or a paper which already took care of this problem. Unfortunately I don't find anything which helps me with my problem. I ...
0
votes
1answer
261 views

Which is better: GPT or RelGAN for text generation?

Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN. So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN I am so ...
2
votes
0answers
32 views

How to prepare the data for text generation task

First, I'm not sure whether the model contains the encoder during training. EOS means end-of-sentence. Encoder and decoder are part of transformer network. If without-encoder, training time: ...
1
vote
1answer
37 views

The principle of LM deep model

Language model(LM) is the task of predicting the next word. Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder. Or both with-...
1
vote
0answers
56 views

NLP to recognize the meaning of a paragraph

How can I apply NLP to extract the summary of a paragraph,I found this but was wondering if there is a better and easy way before implementing it, as the ans seems to be an old one. Basically what I ...
1
vote
0answers
32 views

Clustering and graphing similarities of sentence subjects

I have a bunch of sentences. Each sentence is given a weight of how "close" it is to a particular subject. Ex. "I love reading math books" Subjects for the above sentence = ...
1
vote
1answer
377 views

transfer learning with sentiment analysis?

The question is how good and what are some things to keep in mind when sentiment analysis models are tested on different datasets than they are trained on. Say the task is to perform sentiment ...
2
votes
3answers
131 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
0
votes
0answers
21 views

Hierarchical softmax for prediction

Original paper: https://www.iro.umontreal.ca/~lisa/pointeurs/hierarchical-nnlm-aistats05.pdf I understand that during training (say for text generation), only the probability of the target word (...
4
votes
1answer
101 views

Can CBOW model only accept fixed number of words?

I have a question about CBOW prediction. Suppose my job is to use 3 surrounding words w(t-3), w(t-2), w(t-1)as input to predict one target word w(t). Once the model is trained and I want to predict a ...
2
votes
1answer
21 views

Is there a way to cluster words based on how similarly they sound?

I have a list of words for a fictional world I've made (don't judge lol). My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build ...
1
vote
0answers
28 views

How is maximizing L(lambda1, lamda2, lamda3) equivalent to minimizing perplexity?

In language modeling, L(lambda1, lambda2, lambda3) is defined as: Sum(count of trigram(u,v,w) x q(w|u,v)) where u, v, w are words in the corpus and perplexity ...
1
vote
0answers
59 views

How to extract name of objects from technical description (NLP)

I have a list of technical descriptions of mechanical parts such as Relay Compressor A4u8skk Relay-Start-Compressor sdfvsh2 Relay-Start-Compressor-Copelan 05-569-21 Compressor-J4KS0AK-5gss-fj2 ...
1
vote
0answers
353 views

Word classification (not text classification) using NLP [closed]

I have been trying to extract Person name and Company name out of string. But, I have been facing lot of difficulties. I have a dataset of names and a dataset of company names. In the string, I wish ...
0
votes
0answers
9 views

Replacement for English Wizard

I am using the English Wizard to access the SQL database. Is there any other 3rd party tool similar to the English Wizard that will work on web and desktop application?
1
vote
0answers
54 views

Job Recommendation System

I am building a Job Recommendation System where I have Student Data for different subjects in Machine Learning(Data Viz, Python, Statistics, etc) and their skills from the resume. Need to Recommend ...
1
vote
1answer
477 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
1
vote
2answers
46 views

Continous bag of words claimed to be unsupervised, how is it working?

I'm following these two lectures on CBOW and skip-gram word2vec models. The first is lec 12 and the next lec 13 of a deep learning series https://www.youtube.com/watch?v=syWB-YMYZvI https://www....
1
vote
0answers
25 views

Is there a way to automatically generate a string given an input of another string? [closed]

I am fairly new to data science in general, and I'm wondering if it is possible to use machine learning/natural language processing to generate definitions for various terms. My team has recently ...
1
vote
0answers
30 views

How to learn word embedding from a context on the fly?

Consider the fictional word tahiliuk in the sentence “We found a small, fluffy tahiliuk running around our garden.” While hearing a new word used in context, people are remarkably adept at inferring a ...
0
votes
1answer
25 views

Find correct forms of a word

I'm trying to find all grammatical forms for words from a dataset. I'm new to NLP. I'm using a combo: spaCy (to get a base form of a word), ...
1
vote
0answers
22 views

TextRank algorithm for Web content

I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags ...
0
votes
1answer
20 views

I am getting a Type Error in this Line

Diff = [i - j for i,j in zip(text_features, author_signature)] Diff is a List , text_features = [1, 2, 3] , author_signature = [3, 2, 1]
7
votes
1answer
196 views

Understanding of naive bayes: computing the conditional probabilities

For a task on sentiment analysis, suppose we have some classes represented by $c$ and features $i$. We can represent the conditional probability of each class as: $$P(c | w_i) = \frac{P(w_i|c) \cdot ...
1
vote
0answers
200 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
2
votes
1answer
28 views

Confidence Intervals for Multi-Categorical Votes

I have an ngram-based language model that produces a long tag list for a given sentence. For example, the just-previous sentence, broken into bigrams, and run through the model might produce something ...
1
vote
1answer
230 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
1
vote
0answers
38 views

Using an ontology to recognize named entities in free text

I'm trying to solve a fairly basic problem in NPL efficiently. What tool or software package would you use to identify the words, or group of words that are part of an given ontology within a free ...
1
vote
0answers
19 views

What approach to use to detect violations of media ethics in news?

I've been trying to come up with a solution to detect violations of media ethics in news articles. For example, I need to detect if an article has mentioned the name of a victim of rape. So far, I've ...
2
votes
1answer
166 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...
1
vote
0answers
28 views

Common deep learning practices in NLP for text classification

Are there any articles about best modern techniques in text classification (not only for English texts, but in general)? In particular, I'm interested: what kind of text preprocessing techniques are ...
1
vote
0answers
31 views

Check If Answer is Correct by Similarity

I am new to data science and machine learning. Let's say that I have a question, and some correct answers for that question (for example, 10 correct answers). Is there a way to get a new answer as ...
0
votes
1answer
57 views

Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic?

I need to classify whether a given review or comment is a complaint or appreciation. This is planned to be used in multiple places, product review pages of own site as well as facebook and twitter. ...
2
votes
0answers
24 views

How do I perform Sentiment Analysis on Tweets in the following pattern:

I have tweets obtained based on matches (football) before the match begins. I have tweets which specify a team will win 3-1 and so on which are easily analyzed using regular expressions. I am facing ...
1
vote
0answers
11 views

Grouping prefix and term queries

Consider the following context: You have a website that is offering a search function, which produces a list of matching articles. The search has prefix functionality: if you are looking for "...
1
vote
1answer
36 views

how to encode labels for relationship extraction

I am trying to extract relationship from text. So, lets take the following text "He went to movie. but, they went to school" So, here the relationship's are "He and Movie", "they and school". How ...
1
vote
0answers
164 views

Loss function for Hierarchical Multi-label classification

I am looking to try different loss functions for a hierarchical multi-label classification problem. So far, I have been training different models or submodels (e.g., a simple MLP branch inside a ...
2
votes
0answers
100 views

Joint embedding of word and image

I often see some papers, in which the authors do the point-wise multiplication of word embedding and the image embedding. As the image shows below, my question is how come the implementation works?? I ...
3
votes
0answers
89 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
1
vote
0answers
78 views

How are natural language generation algorithms given a target

I've started learning about NLP and NLG and I'm fascinated! I've been blown away by the things I've seen from NLP; but I have a few questions about NLG. All my questions boil down to this: Given a ...