Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

What are some best Text Representation techniques in NLP

I've studied about various text representation techniques like : Bag of Words, N-gram data modelling, ...
0
votes
0answers
11 views

Building Search Engine using Vector Space Model using a private database

Im trying to build a search engine for a private dataset using vector space model and have encountered following problem. Dataset Dataset is private. It is a collection of unstructed pdf . I have ...
2
votes
2answers
24 views

Document similarity

I have close to 50000 documents in plain text format. Is there a way in which I can group similar documents together? Similarity mostly here is the content similarity. Will transforming the text ...
0
votes
1answer
24 views

How to model for binary classification with 'Phone Call or not' classes?

I have texts as long as 3000 characters and I need to classify these in 'Phone call happened' and 'Phone call didn't happen' classes. The training dataset has a proportion of 58%:42% for Yes and No ...
3
votes
0answers
24 views

NLP problem : Choosing the optimal parser for extracting quantitative values from text

I have a clinical NLP problem for which I would need some help to establish a proper framework. I am trying to extract different elements from echocardiography reports. Those elements are both ...
6
votes
1answer
354 views

How does attention mechanism learn?

I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding. I have this question because I’m ...
1
vote
0answers
18 views

Gensim - Running Similarity Queries example with another query result in low score

I am a newbie to NLP and gensim, and I am trying to run the tutorial (Similarity Queries) (https://radimrehurek.com/gensim/tut3.html). I can follow the example, and run the expected result. However, ...
0
votes
0answers
19 views

Validation loss flattens or increases even after changing model architecture

I am trying to implement the Hierarchical Attention Model on 2.5M Dataset of Amazon Reviews but even though the training loss goes down the validation loss is almost constant. The first thing that ...
0
votes
1answer
18 views

What is the main concept of using lexical,linguistic, semantic or syntactic approach in NLP for cyberbullying

I am really in need of some explanation, I am working on an NLP cyber-bullying detection tool which I will deploy to the web using Django framework, however, am stuck on some idea, can someone explain ...
0
votes
0answers
28 views

Model for keyword extraction in medical texts [closed]

I have a textual data which will be a conversation between doctor and speech-to-text. I'm looking to extract valuable details from that text. Eg.: The patient has abnormal orientation. OR The ...
2
votes
0answers
26 views

Building own embedding from sequence

I have 100 sequences of the word (i.e., action for completing a task). Each of the sequences contains around 350 actions(115 unique actions but all the actions are not used in each sequence. Some of ...
2
votes
1answer
19 views

Find multiple categories of promises in texts

I have conversations a customer with an agent (without punctuation). There are phrases of several categories of promises that an agent gave to a customer (call back, make an appointment, etc.). It has ...
1
vote
0answers
10 views

Tools/tutorials for compiling corpora for NLP experiments?

I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
1
vote
1answer
37 views

Understanding cosine distance with word vectors

I'm a new DL4J user, and I'm running all the works of Shakespeare through a Word2Vec neural net. I've got a pretty basic question about how to understand the results so far. In the below example, ...
1
vote
1answer
30 views

Consider ratings as sentiment labels?

Beginner here! I have a dataset, with reviews of a product as text, ratings for the product. My previous motive was to use Naive Bayes classifier for sentiment analysis. But my data doesn't have the ...
0
votes
1answer
26 views

How to choose the best parameter values for TfidfVectorizer in sklearn library?

Recently, I used TfidfVectorizer in scikit-learn library to calculate a matrix of TF-IDF features. However, I do not know how to set some parameters such as ...
2
votes
1answer
30 views

How to classify named entities of the same type?

I am doing a project where I am extracting date/time entities from text. I'm using a rule-based system to extract the temporal expressions and ground them to an actual date/time. The second part of ...
3
votes
1answer
27 views

Pretrained handwritten OCR model

I've been looking around for pretrained models dedicated to handwritten OCR. So far I've found very little. Could you please share, if you know any? I find tesseract...
2
votes
1answer
49 views

How to validate regex based Resume parser efficiently

I am using rule based logic to extract features from resume. Basically I am trying to find if the candidate switched the company in less than 1 year. So I have the code in place to find it using ...
1
vote
1answer
43 views

Feature extraction from resume using Python without rule based logic

I am working on a resume parser project. Currently, I am using rule-based regex to extract features like University, Experience, Large Companies, etc. So basically I have a set of universities' names ...
1
vote
0answers
12 views

how to train a model to fetch the organization from Resume

I would like to train a model , which can be used to extract the Skills and Organization from CV. Though i have fetched the same manually through regular expression but i want to build a model for ...
1
vote
1answer
19 views

(pre-trained) python package for semantic word similarity

I am searching for a python package that calculates the semantic similarity between words. I do not want to train a model (what most packages seem to offer) - the package should have been pre-trained ...
0
votes
0answers
20 views

detect table of content in PDFs

I have PDF text data that I want to use for NLP tasks. Now during preprocessing, I came to the conclusion that it is not worth analyzing the table of content. It merely generates a bunch of ...
2
votes
2answers
22 views

Question answering (QA) vs Chatbots

Are Question answering (QA) the same as Chatbots? I can not understand the difference between them. For me it's the same thing: interact with a robot that answers questions.
1
vote
1answer
58 views

How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>

As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode ...
1
vote
1answer
19 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
1
vote
0answers
26 views

Meaningful Information retrieval and question answering for unstructured data - Is it even possible?

Hello good NLP people, I am working on a task that gradually seems not solvable for me. My data-set consists of long, messy, unstructured documents (pdfs, doc, docx, scans with tables, graphs, text, ...
1
vote
1answer
15 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
4
votes
2answers
47 views

Removing junk sentences

I have transcripts of phone calls with customers and agents. I'm trying to find promises which were made by an agent to a customer. I already did punctuation restoration. But there are a lot of ...
2
votes
1answer
34 views

What is purpose of the [CLS] token and why its encoding output is important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
1
vote
1answer
30 views

Predicting Missing Word in Text

I know about BERT and other solutions when you masking some words and try to predict them. But let say I have a text: Transformer have taken the of Natural Processing by storm, transforming the ...
2
votes
2answers
28 views

Method to assess text credibility

I am searching for an automated method (ideally a python package) that produces a score to assess the credibility of a given text (e.g. from a webpage). I am not searching for: text complexity ...
1
vote
1answer
33 views

Python package to assess text coherence

I am looking for a python package that calculates how well one sentence of a natural text follows the next. One could simply count how many identical words are in the next sentence but the better ...
0
votes
1answer
24 views

How can I extract the reason of the legal compensation from a court report?

I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report) Order mister X to ...
2
votes
1answer
35 views

Approach to semantic similarity between documents

I was wondering what approach people would take, or point me in the right direction on this challenge I have set myself. I am pretty new at this, I have covered some area but want to expand my ...
2
votes
3answers
62 views

How to approach TF-IDf based analysis?

Problem statement : We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad) docs - ...
2
votes
1answer
28 views

Multilingual Bert sentence vector captures language used more than meaning - working as interned?

Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of [CLS]), then ...
0
votes
0answers
15 views

Extract specific Information from unstructured Documents

I am trying to build a system where recruiter will upload a doc file with Job Roles , Location , Experience , Title . the problem is every user will upload a different format document. Please Visit ...
1
vote
0answers
19 views

How we compare two paragraphs of unlabelled dataset semantically (STS)?

Column representation: Unique_id | Text1 | Text2 Unique_id 0 Text1 public show for reynolds suspension of his coaching licence. portrait sir joshua reynolds portrait of omai will get a public airing ...
2
votes
3answers
38 views

NLP: How to group sub-field into fields?

Suppose I have a list of strings that captures a sub-field of academic research and would like to group them as higher-level fields. For example, ...
8
votes
2answers
117 views

NLP : variations of a text without modifying it's meaning

I am currently working on the automation of recurring reports (weekly 30-50 pages reports for around 100 districts). Those reports have a mostly fixed form : maps, graphs, data tables and small zone ...
1
vote
1answer
19 views

How to incorporate keyboard positions on character level embeddings?

I am working with NLP and have character level embeddings. I have embeddings learned from Wikipedia text. Now, I want to learn embeddings from chat data (where misspellings and abbreviations are way ...
1
vote
0answers
20 views

Lemmatization of pandas column using Wordnet after POS

I have a pandas column df_travail[line_text] with text. I want to lemmatize each word of this column. First I Lowercase the text : ...
1
vote
2answers
33 views

How can we perform STS(Semantic Textual Similarity) on UnSupervised dataset using Deep Learning?

How to implement STS(Semantic Textual Similarity) on unlabelled dataset. Dataset column contains Unique_id, text1(contains paragraph), text2(contains paragraph). Ex: Column representation: Unique_id ...
2
votes
1answer
29 views

Why I get a very low accuracy with LSTM and pretrained word2vec?

I'm working on a reviews classification model with only two categories 0 (negative) and 1 (positive). I'm using pre-trained word2vec from google with LSTM. The problem is I get an accuracy of around ...
2
votes
2answers
18 views

User profiling based on multiple posts

I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each ...
2
votes
2answers
53 views

How to do feature selection after using pre-trained word embeddings?

I am working on a multiclass text classification problem. I want to use the top k features based on mutual information (mutual_info_classif) for training my model. ...
1
vote
0answers
20 views

The differences between BNf and JSGF in NLP?

I wonder what the differences are between the BNF(Backus-Naur Form) and JSGF(Java Speech Grammar Format)? The former is a kind of context-free grammar taught in CS224, but I learned that the JSGF is ...
2
votes
0answers
22 views

Change the way spacy works - Custom properties for training and prediction

Spacy detects the entities using its predefined algorithm. It parses tokens in text considering position of tokens with respect to tokens surrounding it. It also takes into consideration the POS ...
1
vote
0answers
18 views

How to build SAAS Chatbot?

I have a problem which is I'm building a chatbot that is used for some booking systems like event booking, I have built it with rasa and botfront for my personal use. So I'm moving to make it as SAAS ...