Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

Filter by
Sorted by
Tagged with
0
votes
0answers
65 views

What is the State-of-the-Art open source Voice Cloning tool right now?

I would like to clone a voice as precisely as possible. Lately, impressive models have been released that only need about 10 s of voice input (cf. https://github.com/CorentinJ/Real-Time-Voice-Cloning),...
0
votes
1answer
52 views

NMT, What if we do not pass input for decoder?

For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output. What ...
-1
votes
2answers
48 views

What can NLI do for a chatbot?

Natural Language Inference(NLI) is the task of predicting the labels(entailment, contradiction, and neutral,) for sentence pairs. People invent a lot of deep model to solve this problem. But I can ...
2
votes
0answers
16 views

Step extraction from a paragraph

Came across an interesting problem: Given a paragraph describing how to do a process, need to break it down to various steps. Basically, need to determine for each sentence in the paragraph, if this ...
1
vote
1answer
34 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
0
votes
1answer
45 views

Smallest Possible Dataset for Text Classification using BERT

What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis? ~100 examples ~1000 examples ... ~10000000 examples ...
1
vote
0answers
10 views

Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode: ...
1
vote
2answers
53 views

How should I format input and output for text generation with LSTMs

I'm attempting to generate a response to an input line of text using an LSTM. I've considered various forms of input, including one-hot encoding each character in the line and passing each input line ...
1
vote
1answer
75 views

Question classification

I have 10 classes and 10-15 questions in each class . Given a new question, I want to find the class to which the question is most similar?
1
vote
1answer
250 views

How to replace words in a sentence with their POS tag generated with SpaCy efficiently?

How is it possible to replace words in a sentence with their respective PoS tags generated with SpaCy in an efficient way?
0
votes
1answer
63 views

counter vector fit transform cosine similarity memory error

count_matrix = count.fit_transform(off_data3['bag_of_words']) I have count_matrix shape with count_matrix.shape (476147, 482824) ...
0
votes
1answer
19 views

Why are bigger embedding vectors not necessarily better?

I'm wondering why increasing the dimension of a word dimension vector in NLP doesn't necessarily lead to a better result. For instance, on examples I run, I see sometimes that using a pre-trained 100d ...
3
votes
2answers
25 views

Pre-trained models

I am starting off with machine learning so could someone tell if there is some site where one can find the current best performing trained models for any specific problem like sentiment analysis or ...
2
votes
1answer
135 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
1
vote
1answer
36 views

For text match problem, what is the different between question-question match and question-answer match?

I know question-question match is a text similarity problem. What about question-answer or question-doc match? It is used in information retrieval. question-question match is indeed text similarity. ...
0
votes
0answers
5 views

Training data for doc2vec models, general vs specific

I have quite a general question about doc2vec models. Let's say I have a specific NLP task whose goal is to understand the similarity between two sports news articles. Now I have the option to train ...
4
votes
4answers
408 views

Classification of Conversations in Text

I am trying to pick a technique for classifying conversational text. I am concerned about treating the problem at a level of fidelity of each individual message because people often say things like, "...
1
vote
1answer
26 views

Why TREC set two task: document ranking and passage ranking

TREC is https://microsoft.github.io/TREC-2019-Deep-Learning/ I am new to text retrieval. Still can not understand why set the two similar task. Thank you very much.
4
votes
2answers
230 views

What is the best question generation state of art with nlp?

I was trying out various projects available for question generation on GitHub namely NQG,question-generation and a lot of others but I don't see good results form them either they have very bad ...
1
vote
0answers
13 views

How can one determine that Word2Vec (CBOW method) embeddings are related to each other?

I read some fascinating stuff about the potential for using the Word2Vec algorithm to speed up the pace of scientific discovery here https://www.researchgate.net/publication/...
0
votes
0answers
33 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
0
votes
0answers
33 views

Multitask learning in Keras issue

I am implementing multitask regression model using code from the Keras API under the shared layers section. There are two data sets, Let's call them data_1 and <...
0
votes
1answer
28 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
1
vote
0answers
9 views
2
votes
0answers
30 views

pros and cons of lexical vs machine learning methods for text mining

I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic. I have used a simple method of mining documents related to a ...
0
votes
1answer
16 views

Extract Domain related words

I am doing a research regarding on automatic text summarizing. So in order to weighting sentences I need to get words related to a particular field or domain like shown below. ...
1
vote
1answer
118 views

Does it make sense to use TF-IDF to extract most important tokens from a corpus?

I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus. My understanding of TF-IDF is that it is calculated per token per document, so ...
0
votes
0answers
26 views

AWS comprehend service topic modelling takes too much time

I have an AWS Comprehand service in which i created an analysis job for topic modelling.Input to job was just 4.4kB text file, I got correct output after 25-30 minutes, then i tried with 750kB file it ...
0
votes
0answers
11 views

Pretrained model entirely losing its properties

I am a newbie in the field of deep learning, so advance apologies if any mistake is there. I was trying to use pretrained models like BERT and GPT2 for generating language in our native language (say, ...
0
votes
0answers
14 views

What type of model is most appropriate for unsupervised NLP?

I work at a firm where we get lots of client RFQs in various different formats and we're required to format them. Example: Clients ABC sends in: "can I get a mkt in XYZ bond forward settlement 25-jul-...
0
votes
0answers
9 views

How to vectorize unigrams character to use LSH functions?

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate vector ...
0
votes
0answers
13 views

How to set 160 bit vector and assign corresponding ngram

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate ...
2
votes
0answers
35 views

Assigning tags to posts using predefined set of tags

I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. ...
0
votes
0answers
12 views

How can I evaluate out-of-domain question in a domain-specific Q&A bot when I only have in-domain data?

I learned that some popular bots like RASA or LUIS will have "confidence scores" to evaluate the out-of-domain questions, but none of them provide documentation of how they calculate these scores. ...
0
votes
0answers
19 views

How can I recognize if a dot is an abbreviation of a word or the end of a sentence?

I have a text and I need to recognize if the end of a sentence is reached. As dots are used for abbreviations as well I can not do a simple check for a dot. How can I recognize this, maybe with an ...
0
votes
0answers
12 views

How can memory networks perform well in lists/set type?

I was reading this paper about memory networks. As I understood, memory networks can give output in a word. But on Babi dataset's 'list/set' task, its accuracy was almost 80%. What have I ...
0
votes
0answers
15 views

How do I generate text responses that take the context of an input into consideration without the use of defined templates?

tldr: I have pairs of paragraphs (reviews and responses). Given a set of sentences as an input, what are some methods to output appropriate response sentences contingent on the context and sentiment ...
1
vote
0answers
10 views

Predicting class of email based on its associated parameters

I am solving a challenge where the data represented as above is given. In this challenge I have to associate each email with one of the the 2 criteria (automation, Or SEO). Example if email falls ...
0
votes
0answers
22 views

Judging Keywords with Benchmark Dataset

I successfully extract keywords from documents in my corpus in a variety of different ways (like running pagerank on an cooccurance matrix, or textrank, or using a similarity matrix, or generating my ...
1
vote
0answers
76 views

Integration of NLP and Angular application

I'm doing a small POC in which I've trained my Machine Learning model (Naive Bayes) and is saved in ".pkl" (pickle) format. Now my next task is to develop a web application which asks the user to ...
2
votes
2answers
52 views

What are tokens and tokenizations?

I'm a high school senior who is new to data science, and would like to get into natural language processing. I currently know nothing about NLP, and the information online can be overwhelming. What ...
1
vote
1answer
62 views

Categorization of Natural Language Processing Tasks

Problem I am currently learning basics of natural language processing. I see many tasks in this area is assigning labels to each individual words in the sentence, including POS tagging, chunking, ...
2
votes
0answers
23 views

Natural language Generator using Data from table

I am working on some natural language generator part. eg.1 Input to it will be Col1 Col2 Col3 A B 13 X Y 14 Output should be two sentence, one ...
0
votes
0answers
50 views

Reading a visualization of word embeddings

For my Masters Thesis, I created a Word2Vec model. I wanted to show this image to clarify the result. But how does the mapping works to display the words in this 2D space? All words are represented ...
2
votes
2answers
63 views

Need help with entity tagging

I need to design a system which can identify movie and production company names in a sentence. The approach that comes to my ...
0
votes
0answers
21 views

How to byte-pair encode character sequences at prediction time?

Suppose I have a vocabulary V = {'f', 'o', 'b', 'a', 'r', 'fo', 'ob', 'ar', 'foob', 'obar'} from BPE and at prediction time I come across an input ...
0
votes
0answers
14 views

Make top N word predictions using a character model?

Given a character model that can predict (in addition to typical ascii characters) a special end-of-word character, I can make a word prediction by iteratively appending character predictions to my ...
0
votes
1answer
86 views

LSTM input and output for sentiment analysis

I'm studying this LSTM network: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis ...
4
votes
1answer
35 views

How should I treat these non-English documents in the NLP task?

So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a ...
-1
votes
1answer
32 views

How can I categoriese / classify a cluster of words?

I am just wondering if it is possible to classify word clusters? For example if I provide you an array of words [bird,chicken,dock,park,apple,grapes,furits,juice] ...