Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

News articles clustering for ESG and others

Can anyone suggest method to do classify ESG categories. News contains very huge noise so doing direct clustering not working to classify the esg or not in a news. Any idea on this?
0
votes
0answers
6 views

Trying to compress text with NLP

For a university project I need to send text in Spanish via SMS. As these have a cost, I am trying to compress this text in an inefficient way.This consists of first generating a permutation of codes ...
0
votes
0answers
10 views

How to load fine-tuned Electra (TFElectraForSequenceClassification) Model?

I have fine-tuned an Electra Model using the following code. ...
0
votes
0answers
6 views

Autogenerate “responsibilities text” for Job description

I am trying to build a sentence generator for Job description. This I what i want to do: Given a list of words, I want to output a sentence. For eg., Input: list_of_words = ["python","...
1
vote
1answer
15 views

How can I convert text data to CoNLL format?

This is the same question that I posted on stackoverflow, but I wondered stackexchange would be appropriate for this question. I would like to convert text data to CoNLL format. words.txt ...
2
votes
1answer
126 views

How to impute missing text data?

Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values? If it ...
0
votes
1answer
23 views

Why do we fine-tune language models and not just include the data in the pre-training datasets?

One question about the pre-training & fine-tuning process for language models: why is it better to fine-tune using a small dataset rather than including the fine-tuning dataset into the pre-...
0
votes
0answers
6 views

Conference Resolution Web App

I am trying make this web app that takes a sentence such as "Ram is a great boy and he is the topper of his class" and gives out an output as "Ram is a great boy and Ram is the topper ...
0
votes
0answers
7 views

Weird entries in GloVe embeddings causing error

I'm trying to load GloVe embedding data, and when just printing out the words and their corresponding embeddings I get an anomaly. With the following code: ...
0
votes
0answers
4 views

Testing metrics and How to test MLM Bert Models?

I am doing a project with MLM models with bert where i mask parts of sentences and try to predict them. And then i try to check the similarity of the word with the original word. The sentences are ...
0
votes
0answers
12 views

Instagram Profile Similarity Features

I want to find similar IG accounts in a semantic way(not demografic like fan count, language, country,...) and thought of the following features: Post Text Similarity (Embeddings by SBERT, averaging ...
1
vote
1answer
26 views

Handle with very short and very long sequences with Neural Network

I am working on multi-class problem with sequences. My dataset is composed of sequences of data with different length. E.g. 1500 labeled samples: 500 datapoint belongs to class A, 500 class B and 500 ...
0
votes
2answers
13 views

Jargon extraction in a text

I have a big text corpus (documentation from a company) and I want to extract the terms that are specific to that area/business. I can do that using TF or TF-IDF and guide myself by the frequency of ...
2
votes
1answer
131 views

How to classify objects from a description in natural language

My objective is to classify objects that all belong to a certain category, based on a textual description of these objects by humans. My problem is not specific to a certain category of objects, but ...
0
votes
1answer
10 views

NLP behind Google Assistant, Amazon Echo and Apple Siri

How do these assisant based models analyse text and convert them to commands, I mean how do they understand the intent, property and value. I just want to know what are the models used and also I am ...
0
votes
1answer
25 views

How do you research and list up thinkable encoders and decoders when you build NLP models?

I'm a beginner in NLP and deep learning fields, and have stacked with the phase how research and list up available and substitutional encoders and decoders. For example, I read a thesis that ...
0
votes
1answer
13 views

Do I need to read an entire database for a recommendation system?

Let's say I have a database with approx 100000 rows. I want to build a content-based recommendation system. Do I really need to read the entire database to calculate similarity? That would be very ...
0
votes
0answers
11 views

Understanding fastText

fastText is Facebook's open source software to obtain word embeddings (the original paper). Given a document indexed by $n$ and represented by list of n-gram ...
2
votes
1answer
98 views

Interpreting the RGB or HEX value from a description of the color using NLP

We do have models that predict the basic color from its description, by basic color I mean red, blue, black etc. But I would like to develop a model that can spit out the RGB or HEX colors by a ...
0
votes
0answers
27 views

Cross validation in SpaCy NER

I'm working on a custom NER model that I created with SpaCy, and for training/testing purposes I would like to use cross validation. Does SpaCy have the option to somehow perform this? If not, what ...
1
vote
1answer
22 views

Automating 3D Modeling using NLP

I would like to know if there is any way we can automate 3D modeling processes. Like if I give the model a text input such as "create a sphere and give it a red color" and the we need to get ...
2
votes
1answer
23 views

Does spaCy support multiple GPUs?

I was wondering if spaCy supports multi-GPU via mpi4py? I am currently using spaCy's nlp.pipe for Named Entity Recognition on a high-performance-computing cluster ...
0
votes
1answer
31 views

Using NLP to recognize the timeliness of text content

NLP models can classify text content as positive or negative. Except that, we also need to know the timeliness of such text content. That is whether the text is describing something that has already ...
1
vote
1answer
34 views

tf-idf for sentence level features

Many papers mention comparing sentences using the tf-idf metric, e.g. Paper. They state: The first one is based on tf-idf where the value of the the corresponding dimension in the vector ...
0
votes
0answers
21 views

An efficient way to resegment tokenized text into phrases

I have text tokenized on the word level and few lists of phrases stored as tuples. What would be the most efficient way to resegment (and store) the text into phrases? For example, a sentence like: &...
0
votes
0answers
5 views

Genesis most_similar find synonym only (not antonyms)

Is there a way to let model.wv.most_similar in gensim return positive-meaning words only (i.e. that shows synonyms but not antonyms)? For example, if I do: ...
0
votes
1answer
13 views

Document Clustering for given specific clusters in python

How can we classify text in to given specific number of clusters in python? I'm aware that the number of clusters can be specified using some mechanisms like k-means but I need to classify the given ...
0
votes
0answers
8 views

Affinity propagation did not converge, this model will not have any cluster centers

When I try to cluster using affinity propagation, the below error occurs and the number of clusters is one. ...
0
votes
0answers
23 views

A recent survey on unsupervised techniques in NLP [closed]

I'm fairly new to the NLP field, and I was wondering what recent techniques have been employed successfully in NLP in an unsupervised way/without labelled data. I've found a survey, but since I'm new, ...
0
votes
0answers
22 views

Getting Word Embeddings for Sentences using long-former model?

I am new to Huggingface and have few basic queries. This post might be helpful to others as well who are starting to use longformer model from huggingface. Objective: Create Sentence/document ...
0
votes
1answer
17 views

Question regarding training data in word2vec - skip-gram

I have a very simple question regarding the training data in word2vec. In the skip-gram implementation, the training data (if I understand it correctly) is generated as pairs of words like it's shown ...
1
vote
2answers
18 views

Low-dimensional path representation learning

I have a graph (ex: map) and multiple sequences of ids representing different paths. A vertex represents a region/area An edge between 2 vertices : a crossing from a region to another A graph path (...
0
votes
1answer
22 views

WordNetLemmatizer not lemmatizing the word “promotional” even with POS given

When I do wnl.lemmatize('promotional','a') or wnl.lemmatize('promotional',wordnet.ADJ), I get merely ...
0
votes
1answer
28 views

GridSearchCV is Giving me ValueError: number of labels does not match number of samples

I'm trying to run a grid CV parameter search using sklearn.model_selection.GridSearchCV. I keep getting a ValueError that is really confusing me. Below I've included the code for the pipeline I ...
0
votes
0answers
25 views

Novelty prediction Using DBSCAN on “unseen data”

I am trying to build an unsupervised learning model, which will be able to predict outliers on "unseen data." The algorithm I chose is DBSCAN (Density-based spatial clustering of ...
0
votes
1answer
17 views

How does the character convolution work in ELMo?

When I read the original ELMo paper (https://arxiv.org/pdf/1802.05365.pdf), I'm stumped by the following line: The context insensitive type representation uses 2048 character n-gram convolutional ...
0
votes
1answer
9 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
1
vote
1answer
25 views

How to match a corpus with a string of words using a TF-IDF matrix?

I am trying to match strings of words with a website that has bulletpoints whose text is most similar to it. The way I thought of doing it is to get all of the documents from each bulletpoint into one ...
2
votes
1answer
22 views

How to choose and create natural language data for machine learning

What the difference between these two data formats? For example, for the Named Entity Recognition task, I learned that index and BIO Encoding are popular data formats to train. Are they have different ...
0
votes
0answers
35 views

Character Level Embedding in Sentence Classification

I'm working on an NLP task that requires the use of character level embeddings. By using tokenizer library I realized that it tokenizes such as lower integer meant the most frequent character. Is ...
1
vote
0answers
28 views

Hashing trick for dimensionality reduction

I am building a model that uses TF-IDF NLP features in Spark Mllib. The TF-IDF HashingTF function in Mllib uses the 'hashing trick' to efficiently allocate terms to features. My question is: does the ...
0
votes
0answers
11 views

Is there any benchmark dataset for unbalanced text classification?

I want to work on an unbalanced dataset for text classification (sentiment analysis, intent classifier) and hopefully, come up with an idea to improve the classification on such datasets. Is there any ...
0
votes
1answer
22 views

How do the linear + softmax layers give out word probabilities in Transformer network?

I am trying to implement a transformer network from scratch in pytorch to understand it. I am using The illustrated transformer for guidance. The part where I am stuck is about how do we go from the ...
2
votes
1answer
28 views

BERT Optimization for Production

I'm using BERT to transform text into 768 dim vector, It's multilingual : ...
2
votes
1answer
106 views

Number of features of the model must match the input. Model n_features is 740 and input n_features is 400

i am getting this error predicting from random classifier, could anybody point me to where i am going wrong in this? (background information: yes, i am trying to do sentence classification with 2 ...
1
vote
0answers
13 views

How would you utilize NLP for this situation?

I am working on a project that involves pulling data from a free response section of a survey and matching it to one or more pre-determined categories. For example: someone who took the survey could ...
1
vote
1answer
19 views

Document ranking on a web scraped dataset without any labelled data

I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
1
vote
1answer
29 views

One-Class Text Classification

So I have a specific use case where my colleagues have kept thousands of articles across the years deemed as "Good", among hundreds of thousands of other articles deemed as bad and they didn'...
0
votes
1answer
22 views

Kmeans with Word2Vec model unexpected results

I'm trying to play around with unsupervised NLP using Word2Vec. So far, the data i used is very small, but that is because I am just testing to see how Kmeans will work. The Kmeans was performed first ...
1
vote
0answers
41 views

How to extract “defined skills” from texts documents? [closed]

I have a question with automatic natural language processing, I would like to automatically visualize skills (communication, marketing, statistics, etc.) taken from a document. I already have a ...

1
2 3 4 5
40