Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

Filter by
Sorted by
Tagged with
0
votes
1answer
17 views

What does the dimension represent in the GloVe pre-trained word vectors?

I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) to word embedding. I have a conceptual question: What is the difference between these files? On the other hand, what ...
3
votes
1answer
36 views

How to cluster n-grams?

I just wanted to know how to cluster n-grams based on their semantics. Like clustering together n-grams that are semantically similar by leveraging the distributional hypothesis suggesting that ...
0
votes
0answers
6 views

Combining decision trees and neural networks for classifying text with metadata . How to combine and train?

I have a multi-label classification problem where the input consist of free text, with metadata such as categories (from a fixed, limited set) associated with each text. The output consist of a set of ...
0
votes
0answers
10 views

Assign Topic to a document after LDA

I executed my LDA and now I have several topics with their word distributions. How do I assign each document to a topic? Is Euclidean distance a good choice? Or there are other methods? Thanks
0
votes
0answers
10 views

POS extraction using CoreNLP

I have a corpus of windows related documents, for which I need to extract nouns and verbs. However, it is required that I keep certain windows specific words such as "inline hooking", "instruction ...
2
votes
1answer
27 views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
1
vote
3answers
37 views

Using Google Translate API to create a Translation Dataset

Is it a good idea? ;-) Is it legal to do so? Is it legal to release such a dataset to public? Say I have a language X for which I want to create a dataset for translation to/from English, for which I ...
1
vote
1answer
9 views

Automatically categorize parts of a piece of writing

Suppose I had a piece of writing. The document contains aspects like questions, assertions, examples, and explanations. Is it possible to use Natural Language Processing to categorize each sentence of ...
0
votes
0answers
8 views

How to extract electroglottograph/laryngograph using python(specially in “librosa”)?

in a certain project of mine which is related to feature extraction from speech data, I want to extract some electroglottograph/laryngograph from speech data, I have read some research papers but ...
0
votes
0answers
8 views

Where can I get an untokenized version of GLUE's SST-2 dataset?

On the GLUE faq, they say: Similarly, for SST, the data provided is already tokenized. We're working on obtaining a version that is not tokenized. Feel free to train on other distributions of ...
0
votes
0answers
17 views

What is the State-of-the-Art open source Voice Cloning tool right now?

I would like to clone a voice as precisely as possible. Lately, impressive models have been released that only need about 10 s of voice input (cf. https://github.com/CorentinJ/Real-Time-Voice-Cloning),...
0
votes
1answer
17 views

Is it possible to create a rule-based algorithm to compute the relevance score of question-answer pair?

In information retrieval or question answering system, we use TD-IDF or BM25 to compute the similarity score of question-question pair as the baseline or coarse ranking for deep learning. In ...
1
vote
1answer
18 views

What is the easiest way to identify a gender for a noun (in a french)?

I am working on an app where in order to process some data, I need to be able to identify the gender for some selected words. My data is in French. The feature I am looking for should be able to tell ...
0
votes
1answer
37 views

NMT, What if we do not pass input for decoder?

For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output. What ...
2
votes
0answers
16 views

Step extraction from a paragraph

Came across an interesting problem: Given a paragraph describing how to do a process, need to break it down to various steps. Basically, need to determine for each sentence in the paragraph, if this ...
0
votes
1answer
14 views

Identifying if the sentence if it comprise information about education

Given a sentence I am trying to classify if the sentence contain information about education. For example: ...
1
vote
1answer
31 views

How to extract location related terms from raw text in python

I want to extract location related keywords from raw text in python. I have already tried spacy but the results were not good and I just got names of countries while I want fine-grained location ...
0
votes
1answer
15 views

Simplest way to build a semantic analyzer

I want to build a semantic analyzer i.e., to find how similar the meaning of two sentences are. For example- English: Birdie is washing itself in the water basin. English Paraphrase: The bird is ...
0
votes
1answer
29 views

Smallest Possible Dataset for Text Classification using BERT

What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis? ~100 examples ~1000 examples ... ~10000000 examples ...
1
vote
0answers
10 views

Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode: ...
0
votes
1answer
15 views

Why are bigger embedding vectors not necessarily better?

I'm wondering why increasing the dimension of a word dimension vector in NLP doesn't necessarily lead to a better result. For instance, on examples I run, I see sometimes that using a pre-trained 100d ...
0
votes
0answers
5 views

Training data for doc2vec models, general vs specific

I have quite a general question about doc2vec models. Let's say I have a specific NLP task whose goal is to understand the similarity between two sports news articles. Now I have the option to train ...
1
vote
1answer
17 views

Why TREC set two task: document ranking and passage ranking

TREC is https://microsoft.github.io/TREC-2019-Deep-Learning/ I am new to text retrieval. Still can not understand why set the two similar task. Thank you very much.
1
vote
1answer
28 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
-1
votes
2answers
39 views

What can NLI do for a chatbot?

Natural Language Inference(NLI) is the task of predicting the labels(entailment, contradiction, and neutral,) for sentence pairs. People invent a lot of deep model to solve this problem. But I can ...
1
vote
1answer
35 views

For text match problem, what is the different between question-question match and question-answer match?

I know question-question match is a text similarity problem. What about question-answer or question-doc match? It is used in information retrieval. question-question match is indeed text similarity. ...
1
vote
0answers
13 views

How can one determine that Word2Vec (CBOW method) embeddings are related to each other?

I read some fascinating stuff about the potential for using the Word2Vec algorithm to speed up the pace of scientific discovery here https://www.researchgate.net/publication/...
0
votes
0answers
21 views

Multitask learning in Keras issue

I am implementing multitask regression model using code from the Keras API under the shared layers section. There are two data sets, Let's call them data_1 and <...
0
votes
1answer
28 views

Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?
3
votes
2answers
135 views

What is the best question generation state of art with nlp?

I was trying out various projects available for question generation on GitHub namely NQG,question-generation and a lot of others but I don't see good results form them either they have very bad ...
0
votes
0answers
32 views

NLP - How can I encode single words from one feature (No word frequency)

I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of ...
1
vote
0answers
9 views

fit transform with mysterious special chracters

I tried to make the bag of words ...
0
votes
1answer
35 views

counter vector fit transform cosine similarity memory error

count_matrix = count.fit_transform(off_data3['bag_of_words']) I have count_matrix shape with count_matrix.shape (476147, 482824) ...
2
votes
0answers
16 views

pros and cons of lexical vs machine learning methods for text mining

I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic. I have used a simple method of mining documents related to a ...
3
votes
2answers
25 views

Pre-trained models

I am starting off with machine learning so could someone tell if there is some site where one can find the current best performing trained models for any specific problem like sentiment analysis or ...
0
votes
1answer
16 views

Extract Domain related words

I am doing a research regarding on automatic text summarizing. So in order to weighting sentences I need to get words related to a particular field or domain like shown below. ...
0
votes
0answers
22 views

AWS comprehend service topic modelling takes too much time

I have an AWS Comprehand service in which i created an analysis job for topic modelling.Input to job was just 4.4kB text file, I got correct output after 25-30 minutes, then i tried with 750kB file it ...
0
votes
0answers
10 views

Pretrained model entirely losing its properties

I am a newbie in the field of deep learning, so advance apologies if any mistake is there. I was trying to use pretrained models like BERT and GPT2 for generating language in our native language (say, ...
0
votes
0answers
14 views

What type of model is most appropriate for unsupervised NLP?

I work at a firm where we get lots of client RFQs in various different formats and we're required to format them. Example: Clients ABC sends in: "can I get a mkt in XYZ bond forward settlement 25-jul-...
1
vote
1answer
38 views

Emotional tension score in sentences

I am beginner in natural language processing and my goal is to find a way to score sentences based on their emotional tension. More specifically, I would like to know to what degree a sentence ...
0
votes
0answers
7 views

How to vectorize unigrams character to use LSH functions?

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate vector ...
0
votes
0answers
12 views

How to set 160 bit vector and assign corresponding ngram

I would like to implement fuzzy search based on Bloom Filter and LSH hashing. The problem is that: I have found almost ready package to get ngrams from words, now I don't know how to generate ...
2
votes
0answers
35 views

Assigning tags to posts using predefined set of tags

I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. ...
0
votes
0answers
12 views

How can I evaluate out-of-domain question in a domain-specific Q&A bot when I only have in-domain data?

I learned that some popular bots like RASA or LUIS will have "confidence scores" to evaluate the out-of-domain questions, but none of them provide documentation of how they calculate these scores. ...
0
votes
0answers
19 views

How can I recognize if a dot is an abbreviation of a word or the end of a sentence?

I have a text and I need to recognize if the end of a sentence is reached. As dots are used for abbreviations as well I can not do a simple check for a dot. How can I recognize this, maybe with an ...
0
votes
0answers
12 views

How can memory networks perform well in lists/set type?

I was reading this paper about memory networks. As I understood, memory networks can give output in a word. But on Babi dataset's 'list/set' task, its accuracy was almost 80%. What have I ...
1
vote
1answer
81 views

Does it make sense to use TF-IDF to extract most important tokens from a corpus?

I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus. My understanding of TF-IDF is that it is calculated per token per document, so ...
0
votes
0answers
14 views

How do I generate text responses that take the context of an input into consideration without the use of defined templates?

tldr: I have pairs of paragraphs (reviews and responses). Given a set of sentences as an input, what are some methods to output appropriate response sentences contingent on the context and sentiment ...
1
vote
0answers
10 views

Predicting class of email based on its associated parameters

I am solving a challenge where the data represented as above is given. In this challenge I have to associate each email with one of the the 2 criteria (automation, Or SEO). Example if email falls ...
0
votes
0answers
22 views

Judging Keywords with Benchmark Dataset

I successfully extract keywords from documents in my corpus in a variety of different ways (like running pagerank on an cooccurance matrix, or textrank, or using a similarity matrix, or generating my ...