Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
0
votes
0answers
2 views

NLP: Mapping Penn treebank and Brown corpus, to Universal PoS Tags

I am experimenting with NLP and PoS tagging. I wish to build a large corpus, composed of Penn Treebank and Brown corpus, and possibly even more. Unfortunately, their PoS tags are not compatible. Is ...
0
votes
0answers
8 views

I want to know relevant location while scrapping the web either after scrapping or before scrapping

I am making a crawler that monitors social media appearance of various keywords being put by users. So far the problem is getting an irrelevant data sometimes due to location. For example i might need ...
0
votes
1answer
19 views

How to validate a chatbot?

Let's say I'm building a medical assistance chatbot. How do I validate that my model is working well or better than others. Is there any standard validation metrics for it ?
1
vote
0answers
13 views

How do I identify specific parts of a PDF document?

I have a bunch of medical records that I have to input manually. I would like to automate this but all of the records are in different formats. What is the best strategy to build a deep learning model ...
0
votes
0answers
13 views

Training the document page layout and classifying good/bad layouts

I have a use case where I am supposed to get the coordinates of each block element in a page (whether its paragraph, image, table) where I train a model to understand how they are placed in a given ...
1
vote
1answer
17 views

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? That is, to get results which say that, for example, the ...
1
vote
3answers
124 views

Why is data science not yet widely applied to Law? [on hold]

Law (judiciary) contains such a huge corpus to apply NLP to, but yet there are only search engines designed for Law. Why is NLP not yet extensively applied? Is it because of dimensionality?
2
votes
0answers
11 views

How does a CBoW model convert a word to a vector?

A CBOW model actually takes multiple words as inputs and a targeted central word as the output. So, the trained model actually maps several words to a single one, I mean it takes context words and ...
0
votes
1answer
23 views

How to combine nlp and numeric data for a linear regression problem

I'm very new to data science (this is my hello world project), and I have a data set made up of a combination of review text and numerical data such as number of tables. There is also a column for ...
1
vote
2answers
29 views

Where can I find dataset for word analogy task?

In the paper of Word2Vec by Thomas Mikolov and others, there is a accuracy report on the full Semantic-Syntactic data set. Where I can find this dataset or a related dataset for word analogy task? ...
1
vote
1answer
19 views

Is there any way to define custom entities in Spacy

1) I have just started working on NLP the basic Idea is to extract meaningful information from text. For this I am using "Spacy". As far as I have studied Spacy has following entities. ORG PERSON ...
1
vote
0answers
12 views

Document Readability

What are the ways to determine the level of difficulty w.r.t. the complexity of concept for any given document ? I am aware of Flesch Reading Ease Score, but that's generic and based on word count ...
0
votes
0answers
12 views

Why isn't there a Named Entity Recognition task in the GLUE benchmark?

Named Entity Extraction models are important components of Natural Language Understanding systems, for example chatbots. It would be interesting to understand how much the latest state of the art ...
2
votes
1answer
22 views

Is there a way to rank the Extracted Named Entities based on their importance/occurence in a document?

Looking for a way to rank the tens and hundreds of named entities present in any document in order of their importance/relevance in the context. Any thoughts ? Thanks in advance!
-1
votes
0answers
17 views

Difference in approaches of statistical, probabilistic, and machine learning-based parsing [closed]

Existing parsing approaches are basically statistical, probabilistic, and machine learning-based. Please explain between the three approaches. OR refer to the right resources for a beginner like me.
0
votes
0answers
9 views

shallow parsing, dependency parsing, deep parsing

What is the difference between shallow parsing, dependency parsing, deep parsing? I read it on google search that dependency parsing is the bridge between sahallow and deep parsing. Also, the chunking ...
0
votes
1answer
14 views
1
vote
1answer
27 views

NLP Feature creation from phrase matching

I'm building a model to classify email content, to decide whether the email should lead to a JIRA ticket being "Raised" or "Not Raised". The problem I am having is the data is highly imbalanced with ...
0
votes
0answers
5 views

Training data for doc2vec models, general vs specific

I have quite a general question about doc2vec models. Let's say I have a specific NLP task whose goal is to understand the similarity between two sports news articles. Now I have the option to train ...
1
vote
1answer
23 views

Is hyperparameter tuning more affected by the input data, or by the task?

I'm working on optimizing the hyperparameters for several ML models (FFN, CNN, LSTM, BiLSTM, CNN-LSTM) at the moment, and running this alongside another experiment examining which word embeddings are ...
0
votes
0answers
5 views

How to use a ragged tensor with a convolutional layer?

I have textual data of various lengths for which ragged tensors seems well suited. For instance my data could look as follows : ...
0
votes
0answers
8 views

Error setting an array element with a sequence while using TfIdf Vectorirzer and train test split

I am getting value error while trying to classify using TfidfVectorizer. I have looked in the link below but couldn't able to find a solution. [Binary text classification with TfidfVectorizer gives ...
2
votes
0answers
29 views

how to work with NLP with other features

My dataset looks like this ...
0
votes
1answer
17 views

How to properly use approximate_predict() with HDBSCAN clusterer for text clustering (NLP)?

I have approached text clustering using HDBSCAN based on this article which describes how to do this in R. I've re-written this in Python using this library. The approach is to first calculate TF-IDF ...
1
vote
1answer
25 views

Metrics for unsupervised doc2vec model

I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here. The methods provided for checking the quality of the model are very manual and ...
1
vote
1answer
33 views

Is there a good German Stemmer?

What I tried: ...
0
votes
1answer
25 views

How do you integrate nlp into an existing website?

Let's say I have a db driven app which has two tables: people and organizations. I run my documents through a named entity recognition program. Now - what do I do with this additional ner information? ...
2
votes
1answer
17 views

How to segment old digitized newspapers into articles

I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of ...
1
vote
1answer
13 views

Why TREC set two task: document ranking and passage ranking

TREC is https://microsoft.github.io/TREC-2019-Deep-Learning/ I am new to text retrieval. Still can not understand why set the two similar task. Thank you very much.
2
votes
1answer
13 views

Classification of substrings?

What is the appropriate method to find n-grams/sub-phrases/parts-of-sequences that are referring to a specific topic or belong to a certain category? For instance: Imagine a topic of "transfer of ...
1
vote
1answer
23 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
0
votes
0answers
21 views

Sentiment Analysis of News Headlines

I'm trying to do sentiment analysis of News Headlines about a particular subject mentioned in it. Initially, I used TextBlob library for sentiment analysis to ...
0
votes
2answers
30 views

What can NLI do for a chatbot?

Natural Language Inference(NLI) is the task of predicting the labels(entailment, contradiction, and neutral,) for sentence pairs. People invent a lot of deep model to solve this problem. But I can ...
1
vote
1answer
34 views

For text match problem, what is the different between question-question match and question-answer match?

I know question-question match is a text similarity problem. What about question-answer or question-doc match? It is used in information retrieval. question-question match is indeed text similarity. ...
0
votes
1answer
15 views

How to feed data for ngram model?

I want to train an ngram language model Let's say I have the following corpus: ...
1
vote
1answer
40 views

Representation options of strings (keywords/topics) in models

What are all the possible ways to represent keywords in a machine learning model? The two I am aware of are: one hot encoding, using a static index. vector representation, using an embedding layer. ...
1
vote
1answer
26 views

BERT : text classification and feature extractionn

I have tried multi-label text classification with BERT. Here is the sample input: $15.00 hour, customer service, open to industries One of the labels is Billing_rate and prediction score looks ...
0
votes
0answers
20 views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
0
votes
1answer
23 views

Getting Value Error while training a model for binary classification

While training a sequential model using Keras, Im getting this error The model summary is shown below ...
0
votes
0answers
10 views

Can we vectorize nominal text feature using tfidf or count vectorizer?

I recently participated in the hackathon. The dataset includes drug name, sentiments about the drug, unique id. The target variable is the sentiment. It was a sentiment classification or analysis ...
0
votes
0answers
19 views

Summarization of documents using BERT

I have a collection of various documents that are partitioned according to their global topics. For each of these topics, I want to generate a new document that would summarize all of the ...
0
votes
1answer
23 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
1
vote
1answer
16 views

An exhaustive, representative test database in phrase search algorithm

For a phrase searching algorithm, imagine the goal is to search for a name phrase and return matched results based on a pre-defined threshold. For example, searching for "Jon Smith" could return "Jon ...
1
vote
1answer
23 views

In the context of natural language processing, can anyone give a concrete example of True Positive, True Negative, False Positive, False Negative?

Google post gives a interesting explanation about True Positive, True Negative, False Positive, False Negative True Positive (TP): Reality: A wolf threatened. Shepherd said: "Wolf." Outcome: ...
2
votes
2answers
33 views

Which method of NLP is this?

I have been searching for 2 weeks and I got no where so far. There is a list of diseases ...
1
vote
0answers
17 views

Determining topic of text

I was wondering what I should be looking into if I want to measure the similarity between a paragraph and a corpus of text. For example, given a paragraph of text and the entire corpus of Data ...
0
votes
1answer
42 views

Spacy Text classification (Binary Classification)

I have a dataset of two folders. One of them contains the documents(text, pdfs) related to personal information (like name,email,address etc), the other contains non-personal information. I have to ...
0
votes
0answers
8 views

Encoding numbers and words

I am fairly new to seq2seq models in nlp and just really learned about them. Anyway, in many of the examples, I have seen there has been to approaches to providing a model with data. One in which is ...
0
votes
0answers
6 views

What is the impact of <unk> token to quality of machine translation (BLEU)

i've read some papers about machine translation. Authors usually define a threshold to limit vocabulary to minimum rare words and replace rare words into token. So the question is if we increase the ...
1
vote
1answer
26 views

Word2Vec - document similarity

Lets say I have text data for different documents from 2005 - 2015. I want to compare the similarity between $t$ and $t-1$ documents. So I take the document at 2006 and compare it with the document at ...