Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

0
votes
0answers
9 views

Glove supported languages

Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. I started experimenting with words embeddings also, and I found some interesting ...
0
votes
0answers
4 views

Clustering text post vectorization by Universal sentence encoder

I have 1000 of datasets within which clustering needs to happen in each of them to find the textual categories. I have vectorized the data using google sentence encoder. What can be the best way to ...
0
votes
0answers
7 views

QA-Datasets for non-englisch languages?

I wonder if they are communities like for german where you can access german data sets for NLP task. For english there is the QA-dataset Squad. I wonder if such thing exists for german too?
1
vote
0answers
10 views

Contrastive loss problem in a character-level, siamese NN model

Summary: my NN with contrastive loss does not work, need help debugging Background: I am trying to replicate this paper At first, I used binary crossentropy for loss, and the results were very good,...
1
vote
1answer
9 views
4
votes
2answers
20 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
1
vote
0answers
9 views

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python. If I do topic modelling and get different groups of topic, how do I pick out the only topic ...
1
vote
0answers
19 views

Variable input/output length for Transformer

I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site http://jalammar.github.io/illustrated-transformer/ which provided a great ...
1
vote
1answer
13 views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone ...
1
vote
0answers
19 views

Word classification in the context

I'm trying to solve a 'negation-like' classification problem, where I need to classify whether a certain word within the context has negative or positive label. For example, how to identify whether a ...
1
vote
0answers
14 views

Automating scoring of answers for a given question

There is a topic 'X' on which students are asked to write an English passage. Given a question, I have multiple solutions with their scorings out of 10. For a given answer by a new candidate, one ...
0
votes
1answer
15 views

Is there any text similarity databse available for phrases?

I want to train my application for phrase similarity. I want my model to predict similarity score for phrases as shown in below examples. ex- ...
0
votes
0answers
12 views

Long range dependency dataset

I am doing a project concerning formal languages and I want to relate them to DL structures. In this regard, I am looking for datasets that include enough long-range dependency for the use of ...
0
votes
0answers
5 views

Extarct information of of text such as font from pdf using python

I need to extract the information of text from the raw PDF,such as 1.Font size of text 2.Font used Is there any predefined libraries in python to extract the above information
1
vote
0answers
31 views

determining unpronounceability

I would like to determine whether a given string (eg in English) is pronounceable or not , e.g. using soundex or the like . Has anyone run across a way to do this? Examples of unpronounceable: '...
0
votes
0answers
10 views

Convert extracted text into short summary (Abstract summarisation)

I extracted some text from PDF and is in raw format (having lots of text) I need to convert extracted text into meaningful short summary Are there any algorthims available or pre_defined libraries ...
2
votes
1answer
18 views

How to learn irrelevant words in an information retrieval system?

Right now my recommender system for information retrieval uses word embedding stogether with Tfidfs weights like written here: http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-...
1
vote
3answers
66 views

imbalanced dataset in text classififaction

I have a data set collected from Facebook consists of 10 class, each class have 2500 posts, but when count number of unique words in each class, they has different count as shown in the figure Is ...
2
votes
1answer
26 views

prepare email text for nlp (sentiment analysis)

I have text of emails, which also contains disclaimers, phone numbers, email addresses, file attachment names, addresses, greetings etc. At the moment I blindly pass this text through an OOTB ...
1
vote
0answers
9 views

Dealing with missing n-grams in Naive Bayes classifier

I am doing sentiment analysis on code-mixed text data, i.e English used interchangeably with another language. The dataset I currently have is very small in size, approx 3.5k samples. I am sure that ...
0
votes
0answers
10 views

DocumentTerm Matrix Clustering Suggestions

I am trying to use NLP-style techniques to create clusters of similar books based on their "taglines" or "synopses". The problem I am having is that for many of the synopses, only one word is common. ...
0
votes
0answers
15 views

Anomaly detection in text - how to utilize ngram frequency of words for the detection of an anomalous document?

I am calculating ngram frequency on a text, and for each word I output its conditional probability: how frequent it is given thew last n-1 words, or, if you will, p(ngram)/p([n-1]gram). Using this I ...
1
vote
0answers
30 views

How to extract name of objects from technical description (NLP)

I have a list of technical descriptions of mechanical parts such as Relay Compressor A4u8skk Relay-Start-Compressor sdfvsh2 Relay-Start-Compressor-Copelan 05-569-21 Compressor-J4KS0AK-5gss-fj2 ...
2
votes
1answer
29 views

NLP text autoencoder that generates text in poetic meter

I would like to create an NLP autoencoder that happens to only generate text that conforms to a poetic meter, for example 'iambic pentameter'. That is, the output should be a series of clauses which ...
1
vote
0answers
22 views

how to label a tain_data? [closed]

I have one assignment that I have four files 1) train_data.csv: The training file contains two fields (text, id). 2) train_label.csv: The label file contains two fields (id, label). 3) test_data.csv: ...
0
votes
0answers
12 views

define sentences with messy text data

I am extracting text from various file formats: pdf, emails, word docs, text files etc. The raw data will be processed (e.g. stemmed) but it is very likely that there are no clear sentences (e.g. ...
0
votes
0answers
17 views

Identify specify areas in the text

I'd be interested in identifying various areas in the text message. Let's say I have a text containing some introduction, then there is a poem and at the end there are some urls to some web pages. I'...
0
votes
1answer
41 views

Help in NLP Problem

I am trying to solve an NLP problem and, since I am new to NLP I would like to ask for some insight. I have a data-set with 300000 and 14 features. This data set is about customer complaints, and the ...
1
vote
0answers
18 views

What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content [closed]

Think insurance/medical forms that come from different companies. There is no standard on formatting. I am trying to extract the text based on each section of a given form. A form might have a ...
1
vote
1answer
26 views

Find all potential similar documents out of a list of documents using clustering [closed]

I'm working with the quora question pairs csv file which I loaded into a pd dataframe and isolated the qid and question so my questions are in this form : ...
1
vote
0answers
14 views

Latent feature extraction using dnn and word2vec embeddings

I recently read a journal about a tag-aware recommender system. There is a part in the paper which I do not understand. They used word2vec first and using embeddings as input to a DNN to extract ...
3
votes
1answer
44 views

How to train neural word embeddings?

So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then ...
1
vote
0answers
44 views

Word classification (not text classification) using NLP [closed]

I have been trying to extract Person name and Company name out of string. But, I have been facing lot of difficulties. I have a dataset of names and a dataset of company names. In the string, I wish ...
1
vote
0answers
16 views

How does the hashing trick work in NLP

If I have 1000 words and want to reduce the dimension to 500, does that mean per hash there will be two words on average? In the course NLP specialization on courser, 40 million unique word matrix was ...
1
vote
0answers
21 views

sentence patterns/rules/formulas for imperative sentences … [closed]

Please let me ask if you know any ressources on imperative sentence patterns for natural language generation. There’s probably some ressources online, but I did not really find any. Imperative ...
0
votes
0answers
11 views

Verify compliance with the laws in text documents

USE CASE - AS IS A user sends a text document X to a system where it requires a s service (for example, to request a residence permit). A technician examines the document verifying that the document ...
1
vote
1answer
21 views

how to extract the Top contributing labels/words in universal-sentence-encoder-large - TransformerModel?

I'm using the universal-sentence-encoder-large (Transformer Model) encoding process for embedding and then using the embedding for Clustering - Basically for unsupervised learning. I want to get the ...
0
votes
0answers
9 views

Text segmentation based on most probable classes

I am working on a text classification problem. As training data, I have human annotated text, which was manually segmented into sections and then these sections are labeled with some class. Training ...
0
votes
1answer
67 views

How to calculate Accuracy, Precision, Recall and F1 score based on predict_proba matrix?

I found this link that defines Accuracy, Precision, Recall and ...
0
votes
0answers
7 views

Knowing Feature Importance from Sparse Matrix

I was working with a dataset which had a textual column as well as numerical columns, so I used tfidf for textual column and created a sparse matrix, similarly for the numerical features I created a ...
1
vote
1answer
23 views

NLP: What are some popular packages for phrase tokenization?

I'm trying to tokenize some sentences into phrases. For instance, given I think you're cute and I want to know more about you The tokens can be something like I think you're cute and I want ...
0
votes
1answer
33 views

Naive bayes, all of the elements in predict_proba output matrix are less than 0.5

I've created a MultinomialNB classifier model by which I'm trying to label some test texts: ...
0
votes
1answer
10 views

Survey papers on Natural Language Understanding

What are some good survey or review papers on the state of the art of machine learning approaches to grammatical/syntactic structure of natural languages?
0
votes
1answer
29 views

Confidence Score For Trained Sentiment Analyser Model

I have trained a text based sentiment analysis model, using SciKit-learn and custom data. I have the model ready and it works fine in predicting a text to a class (Positive or Negative or Neutral). I ...
1
vote
1answer
33 views

Word embeddings for Information Retrieval - Document search?

What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a ...
0
votes
0answers
26 views

Sentiment analysis with nltk

I'm studying sentimental analysis with python library nltk, following this example: ...
1
vote
1answer
13 views

Transform single-label data set into multi-label data set

I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
0
votes
1answer
27 views

Pretrained word embeddings vs model weights

I'm trying to understand the relationship between pretrained word vectors and pretrained weights when using pretrained word vectors in another neural network. Are the vectors themselves the weights ...
0
votes
0answers
21 views

I want to extract the problem behind the question using NLP/NLU

I want to convert a given question to the problem behind it. For instance, This question "What free software can convert audio files into text files ?" hides a problem/action which is "Convert ...