Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
4
votes
3answers
3k views

How to give name to topics created using LDA?

I have categorized 800,000 documents into 500 categories using the Mahout topic modelling. Instead of representing the topic using the top 5/10 words for each topics, I want to infer a generic name ...
3
votes
1answer
3k views

What are useful evaluation metrics used in machine learning

I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321". All this based on CNN. I want to evaluate my ...
12
votes
1answer
16k views

What is a 1D Convolutional Layer in Deep Learning?

I have a good general understanding of the role and mechanism of convolutional layers in Deep Learning for image processing in case of 2D or 3D implementations - they "simply" try to catch 2D patterns ...
24
votes
3answers
2k views

Why are NLP and Machine Learning communities interested in deep learning?

I hope you can help me, as I have some questions on this topic. I'm new in the field of deep learning, and while I did some tutorials, I can't relate or distinguish concepts from one another.
19
votes
3answers
25k views

How do I load FastText pretrained model with Gensim?

I tried to load fastText pretrained model from here Fasttext model. I am using wiki.simple.en ...
15
votes
2answers
10k views

NLP - Is Gazetteer a cheat?

In NLP, there is the concept of Gazetteer which can be quite useful for creating annotations. As far as I understand: A gazetteer consists of a set of lists ...
16
votes
2answers
17k views

Extract most informative parts of text from documents

Are there any articles or discussions about extracting part of text that holds the most of information about current document. For example, I have a large corpus of documents from the same domain. ...
11
votes
3answers
5k views

Help regarding NER in NLTK

I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ...
7
votes
3answers
3k views

Twitter Sentiment Analysis: Detecting neutral tweets despite training on only Positive and Negative Classes

I am a newbie when it comes to machine learning. I am trying to get hands on experience by analyzing different supervised learning algorithms using scikit-learn library of python. I am using the ...
4
votes
3answers
2k views

Categorizing Customer Emails

I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records ...
4
votes
1answer
1k views

Creating training data

My task is to classify free text originated from customer complaints about our product. I have created a Taxonomy and have around 10 different categories. I've realized that these categories include ...
25
votes
3answers
29k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
27
votes
4answers
24k views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
49
votes
4answers
19k views

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP ...
18
votes
1answer
15k views

Word2Vec vs. Sentence2Vec vs. Doc2Vec

I recently came across the terms Word2Vec, Sentence2Vec and Doc2Vec and kind of confused as I am new to vector semantics. Can someone please elaborate the differences in these methods in simple words. ...
19
votes
3answers
8k views

Dataset for Named Entity Recognition on Informal Text

I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
19
votes
3answers
2k views

How to grow a list of related words based on initial keywords?

I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates ...
12
votes
4answers
4k views

Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats

I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At ...
11
votes
2answers
12k views

How do “intent recognisers” work?

Amazon's Alexa, Nuance's Mix and Facebook's Wit.ai all use a similar system to specify how to convert a text command into an intent - i.e. something a computer would understand. I'm not sure what the "...
19
votes
4answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
17
votes
4answers
7k views

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
11
votes
1answer
2k views

applying word2vec on small text files

I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword ("kw1") and wants to find semantically ...
9
votes
1answer
596 views

How to determine the complexity of an English sentence?

I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small ...
5
votes
2answers
7k views

Text similarity using RNN

Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the ...
3
votes
3answers
252 views

Machine learning or NLP approach to convert string about month ,year into dates

I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates. Example : last year last month into December 2018 string may ...
2
votes
0answers
2k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
9
votes
5answers
10k views

How to create a good list of stopwords

I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and ...
4
votes
1answer
93 views

Can CBOW model only accept fixed number of words?

I have a question about CBOW prediction. Suppose my job is to use 3 surrounding words w(t-3), w(t-2), w(t-1)as input to predict one target word w(t). Once the model is trained and I want to predict a ...
4
votes
1answer
423 views

OpenNLP Coreference Resolution (German)

I need to do coreference resolution for German texts and I plan to use OpenNLP to perform this task. As far as I know OpenNLP coreference resolution does not support the German language. Which ...
4
votes
1answer
2k views

Contributions of each feature in classification?

I have some features and I am using Weka to classify my instances. For example I have: Number of adj number of adverb number of punctuation in my feature ...
3
votes
1answer
2k views

Activation method and Loss function for multilabel multiclass classification

I am using CNN for Sentence Classification code by Yoonkim. This is used for text classification. I noticed that he uses softmax layer and negative log likelihood error. This is optimal for single ...
3
votes
3answers
5k views

Sentiment Analysis model for Spanish

I barely know about Data Analysis tools and techniques, so bare with me if I'm asking something too trivial. I'm looking for a Sentiment Analysis tool to process comments in Spanish. I do know some ...
6
votes
3answers
567 views

Word2Vec how to choose the embedding size parameter

I'm running word2vec over collection of documents. I understand that the size of the model is the number of dimensions of the vector space that the word is embedded into. And that different dimensions ...
5
votes
1answer
534 views

Coreference Resolution for German Texts

Does anyone know a libarary for performing coreference resolution on German texts? As far as I know, OpenNLP and Stanford NLP are not able to perform coreference resolution for German Texts. The ...
3
votes
1answer
67 views

Is it a red flag that increasing the number of parameters makes the model less able to overfit small amounts of data?

I'm training a deep network (CNN-LSTM-CRF) for Named Entity Recognition. Is there a reason that increasing the number of parameters would make the network less able to overfit a small training set (~...
3
votes
1answer
124 views

Can we use Stop Words while using multinomial navies theorem?

I'm collecting Twitter tweets for sentiment analysis. I chose to use Multinomial Navies theorem for finding the sentiment. I found some examples of SVM theorem making use of stop words. My question ...
2
votes
2answers
263 views

Detect related sentences

This question is related to "How to grow a list of related words based on initial keywords?" In the previous question they attempt to get similar words to a given word. However, I am interested in ...
2
votes
1answer
70 views

Text annotating process, quality vs quantity?

I have a question regarding annotating text data for classification. Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time ...
1
vote
1answer
173 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
1
vote
1answer
482 views

Text processing

I am completely new to analyze cluster texts, I'm using Goodreads API to get Books synopsis. My goal is to group similar books, for example: Politics Music Biographies etc... While Goodreads provide ...
1
vote
4answers
1k views

Using HashingVectorizer for text vectorization

Here is the sample data I have: Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P) Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q) Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), ...
1
vote
1answer
28 views

Fewer observations & larger documents vs More observations & smaller documents

Let's suppose that I have a dataset of 1000 documents. Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}. Let's suppose that the ...
1
vote
2answers
103 views

ML model to transform words

I build model that on input have correct word. On output there is possible word written by human (it contain some errors). My training dataset looks that: ...
0
votes
1answer
592 views

How to calculate Accuracy, Precision, Recall and F1 score based on predict_proba matrix?

I found this link that defines Accuracy, Precision, Recall and ...
0
votes
1answer
80 views

Naive bayes, all of the elements in predict_proba output matrix are less than 0.5

I've created a MultinomialNB classifier model by which I'm trying to label some test texts: ...
0
votes
1answer
2k views

Algorithm for classification of words into given categories [closed]

I'm working with textual data from medical field. I have a list of words and I want to build an algorithm that can classify each word into one or more given categories, like Medicine_Name ...