Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

Filter by
Sorted by
Tagged with
20
votes
2answers
3k views

What is the bleu score of professional human translators?

Machine translation models are usually evaluated using bleu score. I want to get some intuition for this score. What is the bleu score of professional human translator? I know it depends on the ...
17
votes
4answers
26k views

Natural Language to SQL query

I have been working on developing a system "Converting Natural Language to SQL Query". I have read the answers from the similar questions, but was not able to get the information that I was looking ...
9
votes
2answers
421 views

what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etc

I am trying to generate an intelligent model which can scan a set of words or strings and classify them as names, mobile numbers, addresses, cities, states, countries and other entities using machine ...
8
votes
2answers
532 views

How prevalent is `C/C++` in machine learning development?

I am currently a data scientist mostly doing NLP, and I do most of my work inPython. Since I didn't get a CS degree in undergrad, I've been limited to very high ...
7
votes
1answer
2k views

Lemmatization Vs Stemming

I have been reading about both these techniques to find the root of the word, but how do we prefer one to the other? Is "Lemmatization" always better than "Stemming"?
7
votes
1answer
5k views

Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to ...
7
votes
1answer
299 views

Why is word prediction an obsession in Natural Language Processing?

I have heard how great BERT is at masked word prediction, i.e. predicting a missing word from a sentence. In a Medium post about BERT, it says: The basic task of a language model is to predict ...
7
votes
1answer
215 views

Understanding of naive bayes: computing the conditional probabilities

For a task on sentiment analysis, suppose we have some classes represented by $c$ and features $i$. We can represent the conditional probability of each class as: $$P(c | w_i) = \frac{P(w_i|c) \cdot ...
6
votes
1answer
901 views

Is it valid to include your validation data in your vocabulary for NLP?

At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, ...
5
votes
3answers
2k views

What is the best question generation state of art with nlp?

I was trying out various projects available for question generation on GitHub namely NQG,question-generation and a lot of others but I don't see good results form them either they have very bad ...
4
votes
1answer
138 views

How should I treat these non-English documents in the NLP task?

So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a ...
4
votes
4answers
722 views

Classification of Conversations in Text

I am trying to pick a technique for classifying conversational text. I am concerned about treating the problem at a level of fidelity of each individual message because people often say things like, "...
4
votes
1answer
125 views

Can CBOW model only accept fixed number of words?

I have a question about CBOW prediction. Suppose my job is to use 3 surrounding words w(t-3), w(t-2), w(t-1)as input to predict one target word w(t). Once the model is trained and I want to predict a ...
4
votes
1answer
151 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
3
votes
2answers
545 views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
3
votes
2answers
527 views

Trying to implement a “smart compose” feature

I found this post on Gmail's smart compose feature, and it got me thinking about trying to implement it myself. https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html The text ...
3
votes
1answer
146 views

How to cluster n-grams?

I just wanted to know how to cluster n-grams based on their semantics. Like clustering together n-grams that are semantically similar by leveraging the distributional hypothesis suggesting that ...
3
votes
0answers
28 views

Dataset availability for automatic text summarization

I'm working on an automatic text summarization NLP problem and looking for a dataset with USA legal case reports similar to the Australian legal case reports dataset in UCI repository. Can you please ...
3
votes
2answers
33 views

Pre-trained models

I am starting off with machine learning so could someone tell if there is some site where one can find the current best performing trained models for any specific problem like sentiment analysis or ...
3
votes
1answer
62 views

How do I visualize data for a natural language processing project?

I am using a question-and-answer dataset. My neural network takes a question and an article content, and outputs where an answer starts (as an integer). To visualize my data, how should I process it ...
3
votes
0answers
121 views

Why does joint embedding of word and images work?

I often see some papers where the authors do point-wise multiplication of word and image embedding (e.g the image below). Why does this implementation works? I do not understand.
3
votes
0answers
100 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
2
votes
3answers
275 views

NLP and one-class classifier building

I have a big dataset containing almost 0.5 billions of tweets. I'm doing some research about how firms are engaged in activism and so far, I have labelled tweets which can be clustered in an activism ...
2
votes
1answer
152 views

Twitter POS and NER: What is state-of-the-art?

What is the current state-of-the-art for pos tagging and named entity recognition for twitter data? Are industrial-strength programs like Spacy and ...
2
votes
2answers
269 views

What are tokens and tokenizations?

I'm a high school senior who is new to data science, and would like to get into natural language processing. I currently know nothing about NLP, and the information online can be overwhelming. What ...
2
votes
2answers
71 views

Need help with entity tagging

I need to design a system which can identify movie and production company names in a sentence. The approach that comes to my ...
2
votes
2answers
3k views

Difference between IOB and IOB2 format?

I have to tag a dataset for NER. I came across conll2002/esp. What I understand so far, in IOB2 format if I want to tag '...
2
votes
2answers
35 views

Creating pronunciation dictionary for ASR

I am working on ASR(automatic speech recoginition) on Somali data as master thesis and now I am stuck with how to create a phonetics or pronunciation dictionary for it. I searched over net and could ...
2
votes
1answer
206 views

Building a search tool and classifying text using NLP and ML

Im a newbie in information Retrieval. Currently Im reading a book entitled "An Introduction to Information Retreival" by Christopher D. Manning and Prabhakar Raghavan. Im trying to implement an ai ...
2
votes
2answers
46 views

What is the easiest way to identify a gender for a noun (in french)?

I am working on an app where in order to process some data, I need to be able to identify the gender for some selected words. My data is in French. The feature I am looking for should be able to tell ...
2
votes
3answers
139 views

Which neural network to choose for classification from text/speech?

I am considering two tasks: Dialog Act Classification from Text (e.g. classify to: question; opinion; ...) Emotion Recognition from Speech (e.g. happy; calm; sad; ...) Which DL model should perform ...
2
votes
1answer
403 views

Fine-tuning NLP models

In computer vision, if we don't have a large training set, a common method is to start with a pre-trained model for some related task (e.g., ImageNet) and fine-tune that model to solve our problem. ...
2
votes
1answer
78 views

Machine Learning and Natural Language Processing : Project Initiation

I am in the research phase of a long project and am willing to get some useful feedback from your side about the most appropriate project path to take. Current situation: A large team of so called ...
2
votes
1answer
1k views

Why are Chunking and IOB tags necessary?

I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions: Why and how is Chunking helpful? Plus ...
2
votes
1answer
23 views

Is there any way to calculate a relevance score between a title and the content of a text?

My question might sound a little bit stupid but I am trying to come up with a way to measure the relevance of the title of a text, let's say a piece of news headline, to the content. My idea would be ...
2
votes
1answer
44 views

Extracting name, date and total from a set of heterogeneous receipts

So, this is how the problem goes: I am trying to extract information from scanned receipts like this, What I have been told is that I would get the textual data from a OCR software, so in short I ...
2
votes
1answer
31 views

How to group chat messages by topic?

I am a newbie in this field. Developer since 20 years and more but never done anything (except tutorials) with ML, DL, and NLP. Though I've already read a bunch of articles and tutorials about this ...
2
votes
1answer
26 views

Automatically categorize parts of a piece of writing

Suppose I had a piece of writing. The document contains aspects like questions, assertions, examples, and explanations. Is it possible to use Natural Language Processing to categorize each sentence of ...
2
votes
1answer
34 views

Is there a way to cluster words based on how similarly they sound?

I have a list of words for a fictional world I've made (don't judge lol). My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build ...
2
votes
1answer
608 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
2
votes
3answers
230 views

What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

I am building a multiclass classifier to predict the "Intent" of a question. There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/...
2
votes
1answer
221 views

word/sentence alignment for English document

I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below: ...
2
votes
2answers
311 views

Training an AI to play Starcraft 2 with superhuman level of performance?

I'm interested in working on challenging AI problems, and after reading this article (https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/) by DeepMind and ...
2
votes
1answer
345 views

GloVe vector representation homomorphism question

In the paper GloVe: Global Vectors for Word Representation, there is this part (bottom of third page) I don't understand: I understand what groups and homomorphisms are. What I don't understand is ...
2
votes
1answer
49 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
2
votes
1answer
51 views

German Chatbot or conversational AI

I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set! So does it make sense to use google translator API to translate the ...
2
votes
0answers
22 views

What could be proper terms for a research direction in natural language processing to measure meaningfulness?

For some time, I did assessments to design metrics on how to recognize well-written and meaningful software requirements. Then I decided to work with Stack Overflow question posts because they are a ...
2
votes
0answers
11 views

Generate Intro-Text for Newsletter

I am trying to implement the following idea. For a daily newsletter I would like to generate an engaging and funny intro text, such as: Good morning. Sorry if there are beer stains and buffalo ...
2
votes
1answer
304 views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
2
votes
0answers
22 views

Step extraction from a paragraph

Came across an interesting problem: Given a paragraph describing how to do a process, need to break it down to various steps. Basically, need to determine for each sentence in the paragraph, if this ...

1
2 3 4 5