Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

Filter by
Sorted by
Tagged with
0
votes
1answer
11 views

Extract Domain related words

I am doing a research regarding on automatic text summarizing. So in order to weighting sentences I need to get words related to a particular field or domain like shown below. ...
0
votes
0answers
10 views

Suggestions for ML/DL projects to solve business problems [closed]

Can someone give me some suggestions for projects that use both machine learning/deep learning to solve business problems? Background: I am a student in Operations Research (aka optimization) and I ...
0
votes
0answers
14 views

AWS comprehend service topic modelling takes too much time

I have an AWS Comprehand service in which i created an analysis job for topic modelling.Input to job was just 4.4kB text file, I got correct output after 25-30 minutes, then i tried with 750kB file it ...
0
votes
0answers
6 views

Pretrained model entirely losing its properties

I am a newbie in the field of deep learning, so advance apologies if any mistake is there. I was trying to use pretrained models like BERT and GPT2 for generating language in our native language (say, ...
0
votes
0answers
12 views

What type of model is most appropriate for unsupervised NLP?

I work at a firm where we get lots of client RFQs in various different formats and we're required to format them. Example: Clients ABC sends in: "can I get a mkt in XYZ bond forward settlement 25-jul-...
1
vote
1answer
33 views

Emotional tension score in sentences

I am beginner in natural language processing and my goal is to find a way to score sentences based on their emotional tension. More specifically, I would like to know to what degree a sentence ...
0
votes
0answers
6 views

How to vectorize unigrams character to use LSH functions?

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate vector ...
0
votes
0answers
10 views

How to set 160 bit vector and assign corresponding ngram

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate ...
2
votes
0answers
31 views

Assigning tags to posts using predefined set of tags

I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. ...
0
votes
0answers
10 views

How can I evaluate out-of-domain question in a domain-specific Q&A bot when I only have in-domain data?

I learned that some popular bots like RASA or LUIS will have "confidence scores" to evaluate the out-of-domain questions, but none of them provide documentation of how they calculate these scores. ...
0
votes
0answers
18 views

How can I recognize if a dot is an abbreviation of a word or the end of a sentence?

I have a text and I need to recognize if the end of a sentence is reached. As dots are used for abbreviations as well I can not do a simple check for a dot. How can I recognize this, maybe with an ...
0
votes
0answers
11 views

How can memory networks perform well in lists/set type?

I was reading this paper about memory networks. As I understood, memory networks can give output in a word. But on Babi dataset's 'list/set' task, its accuracy was almost 80%. What have I ...
1
vote
1answer
41 views

Does it make sense to use TF-IDF to extract most important tokens from a corpus?

I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus. My understanding of TF-IDF is that it is calculated per token per document, so ...
0
votes
0answers
11 views

How do I generate text responses that take the context of an input into consideration without the use of defined templates?

tldr: I have pairs of paragraphs (reviews and responses). Given a set of sentences as an input, what are some methods to output appropriate response sentences contingent on the context and sentiment ...
1
vote
0answers
10 views

Predicting class of email based on its associated parameters

I am solving a challenge where the data represented as above is given. In this challenge I have to associate each email with one of the the 2 criteria (automation, Or SEO). Example if email falls ...
0
votes
0answers
22 views

Judging Keywords with Benchmark Dataset

I successfully extract keywords from documents in my corpus in a variety of different ways (like running pagerank on an cooccurance matrix, or textrank, or using a similarity matrix, or generating my ...
1
vote
0answers
26 views

Integration of NLP and Angular application

I'm doing a small POC in which I've trained my Machine Learning model (Naive Bayes) and is saved in ".pkl" (pickle) format. Now my next task is to develop a web application which asks the user to ...
1
vote
1answer
27 views

Framing Sentences based on keywords

Given a few specific words, which techniques of Natural Language Processing can I use to achieve creating a meaningful sentence from those words? eg. Words: jackets, highest sale, sweaters, lowest ...
3
votes
1answer
39 views

How do I visualize data for a natural language processing project?

I am using a question-and-answer dataset. My neural network takes a question and an article content, and outputs where an answer starts (as an integer). To visualize my data, how should I process it ...
2
votes
2answers
30 views

What are tokens and tokenizations?

I'm a high school senior who is new to data science, and would like to get into natural language processing. I currently know nothing about NLP, and the information online can be overwhelming. What ...
1
vote
1answer
23 views

Categorization of Natural Language Processing Tasks

Problem I am currently learning basics of natural language processing. I see many tasks in this area is assigning labels to each individual words in the sentence, including POS tagging, chunking, ...
2
votes
0answers
20 views

Natural language Generator using Data from table

I am working on some natural language generator part. eg.1 Input to it will be Col1 Col2 Col3 A B 13 X Y 14 Output should be two sentence, one ...
1
vote
1answer
38 views

How to replace words in a sentence with their POS tag generated with SpaCy efficiently?

How is it possible to replace words in a sentence with their respective PoS tags generated with SpaCy in an efficient way?
0
votes
0answers
21 views

Reading a visualization of word embeddings

For my Masters Thesis, I created a Word2Vec model. I wanted to show this image to clarify the result. But how does the mapping works to display the words in this 2D space? All words are represented ...
0
votes
1answer
68 views

What is the use of [SEP] in paper BERT?

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. However, I have a question. If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
0
votes
0answers
40 views

How to decide between using machine learning and deep learning for a NLP classification problem?

I have a classification problem at hand for which I tried out some traditional ML models to get a glimpse of what I get as results. Brief description of the BINARY classification problem: I have a ...
0
votes
0answers
12 views

Make top N word predictions using a character model?

Given a character model that can predict (in addition to typical ascii characters) a special end-of-word character, I can make a word prediction by iteratively appending character predictions to my ...
0
votes
1answer
39 views

LSTM input and output for sentiment analysis

I'm studying this LSTM network: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis ...
4
votes
1answer
33 views

How should I treat these non-English documents in the NLP task?

So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a ...
0
votes
1answer
20 views

Using Keras how and what do I need to export to use my classifier independently?

I have a basic question that I can't seem to find an answer to. I built and trained with good results (above 90% accuracy) a NLP Log classifier that takes in a UTF-8 payload and classifies it into 32 ...
2
votes
2answers
58 views

Need help with entity tagging

I need to design a system which can identify movie and production company names in a sentence. The approach that comes to my ...
-1
votes
1answer
28 views

How can I categoriese / classify a cluster of words?

I am just wondering if it is possible to classify word clusters? For example if I provide you an array of words [bird,chicken,dock,park,apple,grapes,furits,juice] ...
0
votes
1answer
76 views

*Challenge* Making an algo that learns from a book, and can answer anything about it

I recently took this challenge where I am trying to make a set of algorithms to read any particular book, understand and store the context and subsequently answer any question asked about it. In ways ...
7
votes
1answer
696 views

Lemmatization Vs Stemming

I have been reading about both these techniques to find the root of the word, but how do we prefer one to the other? Is "Lemmatization" always better than "Stemming"?
0
votes
0answers
20 views

How to byte-pair encode character sequences at prediction time?

Suppose I have a vocabulary V = {'f', 'o', 'b', 'a', 'r', 'fo', 'ob', 'ar', 'foob', 'obar'} from BPE and at prediction time I come across an input ...
0
votes
0answers
23 views

How to do Natural Language Processing with few samples only?

I am familiar with SMOTE (Synthetic Minority Oversampling), and the Python Library imbalanced-learn, which can be used to handle ...
2
votes
0answers
69 views

How does BERT deal with catastrophic forgetting?

In the ULMFit paper authors propose a strategy of gradual unfreezing in order to deal with catastrophic forgetting. That is, when the model starts be fine-tuned according to a downstream task, there ...
0
votes
0answers
15 views

Are these Multi-label document classification experiment steps sensible?

I plan to filter an input document using 4 different labels. Just for an example, a document discussing about movie summary needs to be labeled with 4 labels (Romance, Drama, Fiction, Hollywood). ...
0
votes
0answers
25 views

How can I find colours in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
0
votes
1answer
22 views

How to cluster text-based software requirements

I'm beginner in deep learning and I'd like to cluster text-based software requirements by themes (words similarities/frequency of words) using neural networks. Is there any example/tutorial/github ...
1
vote
1answer
36 views

Why Heaps' Law Equation looks so different in this NLP course?

I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed ...
1
vote
0answers
108 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
1
vote
0answers
36 views

Implementation of NLP to categorize text into two categories

I can't discuss my actual dataset, so please bear with me. Let's say I have a dataset that contains a population of 20,000 examinations by a school principal. The principal is to record their ...
0
votes
0answers
23 views

Given an input phrase, is there a way I can find the most similar phrase within a document?

I am completing a task where I need to retrieve the corresponding values to a set of given labels from many legal contracts. For example, one of the labels is "Floating rate payment dates" and it's ...
1
vote
0answers
15 views

Why is MLP working similar to RNN for text generation

I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus ...
1
vote
1answer
97 views

What are CRF (Conditional Random Field)

Looking for language modeling, I have been finding CRF in a lot of places which is but looking online for the same isn't actually helping me a lot. I referred Edwin Chen's blog and Ravish Chawala's ...
0
votes
1answer
47 views

How to calculate which word fits the best given a context and possible words?

I have this task for research purposes and searched a while for a framework or a paper which already took care of this problem. Unfortunately I don't find anything which helps me with my problem. I ...
0
votes
1answer
119 views

Which is better: GPT or RelGAN for text generation?

Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN. So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN I am so ...
2
votes
0answers
30 views

How to prepare the data for text generation task

First, I'm not sure whether the model contains the encoder during training. EOS means end-of-sentence. Encoder and decoder are part of transformer network. If without-encoder, training time: ...
1
vote
1answer
34 views

The principle of LM deep model

Language model(LM) is the task of predicting the next word. Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder. Or both with-...