Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

901 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9 votes
2 answers
274 views

How to implement hierarchical labeling classification?

I am currently working on the task of eCommerce product name classification, so I have categories and subcategories in product data. I noticed that using subcategories as labels delivers worse results ...
user avatar
  • 161
6 votes
0 answers
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
user avatar
  • 676
5 votes
1 answer
909 views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
user avatar
  • 151
5 votes
1 answer
554 views

How to use multiple text features for NLP classifier?

I am trying to build text classifier, Usually, we have one text column and ground truth. But I am working on a problem where dataset contains many text features. I am exploring different ways how to ...
user avatar
5 votes
1 answer
71 views

Is there a way to rank the Extracted Named Entities based on their importance/occurence in a document?

Looking for a way to rank the tens and hundreds of named entities present in any document in order of their importance/relevance in the context. Any thoughts ? Thanks in advance!
user avatar
  • 61
5 votes
2 answers
338 views

Changing default values of ANNIE resources in GATE from Java code

In GATE, default values for ANNIE are set during initialization, but sometimes based on requirements they have to be changed. My Requirement : I want to extract English sentences without considering ...
user avatar
  • 161
4 votes
1 answer
53 views

averaging multiple scores on small chunks of data or raw score on single collated data

I am using IBM Watson tool to determine tones (https://tone-analyzer-demo.mybluemix.net/) and personality scores (https://personality-insights-livedemo.mybluemix.net/) on different files containing ...
user avatar
  • 41
3 votes
1 answer
24 views

NLP text representation techniques that preserve word order in sentence?

I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
user avatar
  • 359
3 votes
0 answers
31 views

Understanding Kneser-Ney Formula for implementation

I am trying to implement this formula in Python $$ \frac{\text{max}(c_{KN}(w^{i}_{i-n+1} - d), 0)}{c_{KN}(w^{i-1}_{i-n+1})} + \lambda(c_{KN}(w^{i-1}_{i-n+1})\mathbb{P}(c_{KN}(w_{i}|w^{i-1}_{i-n+2})$$ ...
user avatar
  • 237
3 votes
0 answers
110 views

Medical NER for French language

I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
user avatar
  • 285
3 votes
3 answers
1k views

Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
user avatar
3 votes
2 answers
355 views

Detecting grammatical errors with BERT

We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
user avatar
  • 285
3 votes
1 answer
85 views

What is the structure and dimension of input passed to neural network when training CBOW and SKIP GRAM word embedding

I am confused about input passed to neural network in natural language processing (NLP) when training CBOW word embedding from scratch. I read the paper and have ...
user avatar
  • 257
3 votes
1 answer
196 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we are adding 1 to each word in our vocabulary, but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me ...
user avatar
  • 113
3 votes
2 answers
646 views

Automatic topic labelling for topic modelling

I am just curious to know if there is a way to automatically get the lables for the topics in Topic modelling. It would be really helpful if there's any python implementation of it.
user avatar
3 votes
2 answers
114 views

Sentiment analysis of tweets (Train model on a labelled dataset and use on some other unlabelled data)

I have a huge amount of tweets on a particular topic say 'ABC' and the data is not labelled. I want to perform multi-class sentiment analysis of these tweets. I tried many unsupervised clustering ...
user avatar
3 votes
0 answers
32 views

Dataset availability for automatic text summarization

I'm working on an automatic text summarization NLP problem and looking for a dataset with USA legal case reports similar to the Australian legal case reports dataset in UCI repository. Can you please ...
user avatar
  • 359
3 votes
1 answer
224 views

FastText Model Explained

I was reading the FastText paper and I have a few questions about the model used for classification. Since I am not from NLP background, some I am unfamiliar with the jargon. In the figure, what ...
user avatar
3 votes
1 answer
321 views

Character-level embeddings in python

I'm working on an NLP task that requires the use of character level embeddings, and I've been trying to use Spacy. However, it seems that spacy uses word-level embeddings for the word vectors, and I ...
user avatar
  • 163
3 votes
0 answers
26 views

Information Extraction from image / text - approach?

I need assistance with a ML project I am currently trying to create. I receive a lot of invoices from a lot of different suppliers - all in their own unique layout. I need to extract 3 key elements ...
user avatar
  • 175
3 votes
1 answer
899 views

find bigrams in pandas

I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
user avatar
  • 71
3 votes
0 answers
48 views

Change the way spacy works - Custom properties for training and prediction

Spacy detects the entities using its predefined algorithm. It parses tokens in text considering position of tokens with respect to tokens surrounding it. It also takes into consideration the POS ...
user avatar
3 votes
1 answer
146 views

Google NLP AutoML

I am doing research for Google NLP AutoML, What methodologies they have used, techniques, models, feature selection, hyper parameter optimization, etc. I could not find any paper on how google built ...
user avatar
  • 497
3 votes
4 answers
70 views

Pre-trained models

I am starting off with machine learning so could someone tell if there is some site where one can find the current best performing trained models for any specific problem like sentiment analysis or ...
user avatar
3 votes
0 answers
28 views

Back-Translation model for German and English

Do you know of any pre-trained models for back translation between German and English? I am aware that there are ways to include a monolingual corpus into the training of a machine translation model (...
user avatar
3 votes
1 answer
84 views

How do I visualize data for a natural language processing project?

I am using a question-and-answer dataset. My neural network takes a question and an article content, and outputs where an answer starts (as an integer). To visualize my data, how should I process it ...
user avatar
3 votes
1 answer
411 views

Convert natural language text to structured data

Convert natural language text to structured data. I'm developing a bot to help user assist in identifying Apparels. The problem is to convert natural language text to structured data (list of ...
user avatar
  • 31
3 votes
2 answers
122 views

How to train neural word embeddings?

So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then ...
user avatar
3 votes
1 answer
385 views

How to train millions of doc2vec embeddings using GPU?

I am trying to train a doc2vec based on user browsing history (urls tagged to user_id). I use chainer deep learning framework. There are more than 20 millions (user_id and urls) of embeddings to ...
user avatar
  • 171
3 votes
1 answer
135 views

How to fetch text from pdf to further proceed with question answer based model from the same document?

To illustrate the above title. Suppose you have a pdf document, which is basically scanned from hardcopy, now there are set of fixed questions to answer from the document itself. For an example a ...
user avatar
3 votes
1 answer
151 views

Why does joint embedding of word and images work?

I often see some papers where the authors do point-wise multiplication of word and image embedding (e.g the image below). Why does this implementation works? I do not understand.
user avatar
3 votes
0 answers
108 views

multiple intents for modifying an intent of a sentence?

Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), ...
user avatar
3 votes
1 answer
74 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
user avatar
  • 31
3 votes
0 answers
178 views

How does Api.ai Google dialogueflow classifies "intent" and extracts data from slots

I am trying to build a very naive version of Api.ai, now Google DailogueFlow. I wanted to know two things. How DF classifies sentences with entities in it that can be user created and/or things like ...
user avatar
  • 31
3 votes
0 answers
381 views

sLDA vs. LDA+Classifier

For simplicity, suppose we're looking at Yelp reviews of restaurants, and are trying to classify the restaurant by cuisine type (e.g. "Italian, Japanese," etc.). Lets also assume our data already a ...
user avatar
  • 261
3 votes
1 answer
572 views

What is the state of the art method for synonym detection?

In Natural Language Processing and Computational Linguistic what methods are deemed as SOA for similar word extraction? Can anyone direct me to those resources?
user avatar
  • 341
3 votes
1 answer
468 views

Complete a Hungarian stem to a real word

I'm quite new to the NLTK package of Python and to NLP too (I usually work in R but for NLP purposes and scraping maybe Python is more able). I scrap articles from Hungarian newsportals and want to ...
user avatar
  • 820
3 votes
0 answers
301 views

How does the supposed "Unified Architecture for NLP" from Collobert and Weston 2008 really works?

In this paper (here) they suppose a "unified architecture for NLP" with deep neural networks with multitask learning My problem is to understand the layered architecture in figure 1, see ...
user avatar
3 votes
2 answers
365 views

Is NLP suitable for my legal contract parsing problem?

My company has a product that involves the extraction of a variety of fields from legal contract PDFs. The current approach is very time consuming and messy, and I am exploring if NLP is a suitable ...
user avatar
2 votes
0 answers
10 views

NLP : What is the difference between Authorship Attribution, Authorship Identification and Authorship Recognition?

I have to write my Master's thesis on this topic (I'm in Natural Language Processing) and while I sometimes see these terms used interchangeably other sources seem to emphasize the fact that there are ...
user avatar
2 votes
0 answers
13 views

Looking for a generalized (extended) lemmatizer

Whenever I lemmatize a compound word in English or German, I obtain a result that ignores the compound structure, e.g. for 'sidekicks' the NLTK WordNet lemmatizer returns 'sidekick', for '...
user avatar
2 votes
1 answer
30 views

how to deal with large numbers of unlabelled target dataset?

I have dataset of 5000 jobs descriptions out of which only 200 jobs are labelled with required English level score range between 0 to 9 and I want to predict remaining 4800 jobs required English level ...
user avatar
2 votes
0 answers
28 views

Perplexed by perplexity

I've seen 2 definitions of the perplexity metric: $PP = 2^{H(p)}$ and $PP = 2^{H(p, q)}$ If I'm understanding correctly, the first one only tells us about how confident the model is about its ...
user avatar
2 votes
0 answers
25 views

skip gram vector representation

I am using SVM for sentiment analysis project , I want to use n-skip gram for vector representation because I need to keep the semantic meaning between words , so I need to make vectors that belongs ...
user avatar
2 votes
0 answers
16 views

Multi-Class Document Classification with both known and un-known classes

Currently, I am building a multi-class document classifier which has to classify either 3 known classes, namely "Financial Report", "Insurance_Sheet", "Endorsement", and ...
user avatar
2 votes
1 answer
28 views

Learning to Rank with Unlabelled Dataset

I have folder of about 60k PDF documents that I would like to learn to rank based on queries to surface the most relevant results. The goal is to surface and rank relevant documents, very much like a ...
user avatar
  • 51
2 votes
0 answers
29 views

Cluster tabular data with text in some columns

Let's say I have a following features in the my dataframe: user_id user_age is_student is_graduate salary resume integer integer binary binary integer text (up to 1000 symbols) And also a few more ...
user avatar
  • 21
2 votes
0 answers
18 views

Classification Texte with naive bayes complement

Currently I am on a text classification project, the goal is to classify a set of CVs according to 13 classes. I use the bayes algorithm (ComplementNB), in my tests it is the model that gives the ...
user avatar
2 votes
0 answers
18 views

Comparing the cosine similarities of the same word representations, from two separate models (vector spaces)

I am comparing the cosine similarities of word representations derived from a BERT model and also from a static Word2Vec model. I understand that the vector spaces of the two models are inherently ...
user avatar
2 votes
0 answers
69 views

Ways to cluster word senses with word embeddings

I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the ...
user avatar

1
2 3 4 5
19