Questions tagged [natural-language-process]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

Should I keep common stop-words when preprocessing for word embedding?

If I want to construct a word embedding by predicting a target word given context words, is it better to remove stop words or keep them? the quick brown fox jumped over the lazy dog or quick brown ...
0
votes
0answers
19 views

How to insert sentences into training data which has 2 words, 3 words 4 and 5 etc into training data?

Hi i have set of sentences where it has 2 words, 3 words, 4 words ,5 words etc. When I'm trying to give train data only 1st two words in sentences it's not accepting. It's showing given 4 expected 2. ...
0
votes
1answer
20 views

Prediction using words which were not in training in a CNN with pre-trained word embeddings

In sentence classification using pre-trained embeddings(fasttext) in a CNN, how does the CNN predict the category of a sentence when the words were not in the training set? I think the trained model ...
0
votes
1answer
18 views

I tried loading my saved .h5 model and predicting with that model, i'm getting error list index out of range

I saved my keras model into .h5 format. Again I've loaded that .h5 file into my colab and tried to predict with that model. ...
0
votes
1answer
24 views

how to train custom word2vec embeddings to find related articles?

I am beginner in machine learning. My project is to make search engine based on AI which shows related articles when we search on website. For this i decided to train my own embedding. I found two ...
1
vote
0answers
18 views

Semantic networks and conceptual graphs

I would like to use semantic networks to understand changes in texts. For example when I add/remove some words within a text. ...
1
vote
1answer
16 views

How can I compare the grammatical complexity between two texts using their sentences dependency length?

This is a continuation to the following thread. I have two texts, common English texts such as news articles and informative texts versus a technical textbook. I want to compare the grammatical ...
0
votes
0answers
18 views

Does GPT-2 has pre trained for sentiment analysis?

I tried sentiment analysis with 345M model of GPT-2. But it took a long time to train. So is there any other GPT-2 model available for sentiment analysis?
1
vote
1answer
34 views

Creating a valid dataset for obtaining results

I have created a domain-specific dataset, lets say it is relating to python programming topic posts. I have taken data from various places specific to this topic to create positive examples in my ...
2
votes
1answer
152 views

Twitter POS and NER: What is state-of-the-art?

What is the current state-of-the-art for pos tagging and named entity recognition for twitter data? Are industrial-strength programs like Spacy and ...
0
votes
0answers
5 views

Is there any inductive Graph Variational Auto Encoder?

I have been reading about how we can model a Variational AutoEncoder (VAE) into a Graph Variational AutoEncoder (GVAE) where the decoder reconstructs the adjacency matrix. I presume that this approach ...
0
votes
0answers
33 views

Semantic network using word2vec

I have thousands of headlines and I would like to build a semantic network using word2vec, specifically google news file. My sentences look like ...
0
votes
0answers
7 views

Multi class product classification according to description of product

I have some products, along with their description. I wish to assign USPSC code to each product. This would require classification on 4 levels. All the examples I could find online were that of ...
0
votes
1answer
12 views

Product classification according to description

I have some products, along with their description. I wish to assign USPSC code to each product. I have a really basic doubt here. What exactly is my test file and training file? Eg. Should the ...
1
vote
0answers
15 views

How to do embedding for nested dictionary with varying size?

I'm working on an RL task in which the agent needs have some observation. Instead using images, I want to use available information of the environment as the observation. The information regarding the ...
0
votes
0answers
9 views

Building quick web app to deploy ML models [migrated]

I've almost completed my DL models for an NLP project. Now I want to make a web app. I created models in PyTorch to detect propaganda text fragments in news articles. The models have poor performance. ...
2
votes
1answer
49 views

KeyError: Selecting text from a dataframe based on values of another dataframe

I have the following two dataframes badges and comments. I have created a list of 'gold users' from ...
0
votes
0answers
9 views

Retrieving information from 2 or more approaches

I'm trying to extract information from documents. There are two approaches currently which produces the following cases where both approaches ...
0
votes
1answer
37 views
1
vote
1answer
16 views

HMM and its competitive alternatives

In Natural language processing, what are the major applications of Hidden Markov Chain (HMM), and what are the alternatives that usually can outperform HMM, is RNN and LSTM always the choice right now?...
0
votes
1answer
27 views

How to identify similar words as input from a dictionary

Let's say I have a CSV file (single column) with a list of words as input. Meaning the file looks like below Input Terms ...
3
votes
0answers
28 views

Dataset availability for automatic text summarization

I'm working on an automatic text summarization NLP problem and looking for a dataset with USA legal case reports similar to the Australian legal case reports dataset in UCI repository. Can you please ...
0
votes
0answers
14 views

Is there a way to find difference between topics in two languages using nlp

I want to analyse queries and their differences between two different languages English and Spanish in this case. I'm aware about topic modelling. I'm in search of any corpus available or any ...
0
votes
0answers
14 views

Looking for a web based tool for natural language to PostgreSQL query for custom database

I need to build a web-based solution that will take a natural language query in English, convert it into an SQL query that I need to execute on a PostgreSQL database. I tried ln2sql/nl2sql but it only ...
1
vote
2answers
50 views

NLP: Compare tags semantically with machine learning? (finding synonyms)

Let's say I have multiple tags that I need to compare semantically. For example: ...
1
vote
2answers
50 views

Where can I get Insurance claim data for practicing NLP(Natural Language) processing?

I am looking for specifically Insurance dataset for practicing Machine Learning & NLP, but unable to find much in kaggle, udemy or other websites. Is there a way to get that dataset or any website ...
0
votes
1answer
20 views

What is syntax V and S standing for nominal subject?

I was reading the recent paper https://www.aclweb.org/anthology/P19-1580.pdf and noticed that in section 5.2, the syntactic relation is studied in terms of the "direction between two tokens". In table ...
1
vote
1answer
30 views

How to figure out if two sentences have the same meaning with AI?

I have two sentences which might be similar in meaning. Are there a useful and successful (machine learning) algorithms, which is able to determine the semantic similarity? Are there any approaches ...
2
votes
1answer
51 views

German Chatbot or conversational AI

I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set! So does it make sense to use google translator API to translate the ...
1
vote
2answers
54 views

Extract information using NLP and store it in csv file

I have a text file that stores the pickup, drops, and time. SMS text is a dummy file that is used to train a cab service model. The text is like in this format: ...
0
votes
1answer
50 views

How to use fine tuning of BERT when i have unlabelled dataset of text documents?

I have gained a basic understanding of using BERT for various NLP/text mining tasks. When it comes to fine-tuning of BERT, I always see that fine-tuning is performed using some classification tasks. ...
1
vote
0answers
18 views

Natural language processing for ecommerce search flow

I'm implementing an ecommerce search using nlp. there is a lack of clear documentation anywhere online about implementation and flow. this is what i'm doing. Search query comes in Do spellcheck Do ...
1
vote
1answer
17 views

Generate text using user-supplied keywords

I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need: User input: End-User: Data Scientists Region: Middle East ...
0
votes
0answers
8 views

How to classify very short text for spend analytics?

I have a very small description of the item, example below- ACCELATOR SHAFT ADAPTOR –EGR PIPE & EGR VALVE(125KVA) ADAPTOR-EGR COOLER AND VALVE-125 KVA ADJUSTING LATCH - ALTERNATOR-125 KVA Some ...
0
votes
0answers
11 views

Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
0
votes
0answers
24 views

Natural Language Processing: Identifying Words That Are Out of Place?

I want to make it easier to manually (or potentially automatically) correct AI transcription. I've noticed that one significant error that can be very easily picked up by human transcriptionists is ...
1
vote
0answers
30 views

Removing duplicate records before training

I am currently working on a project classifying text into classes. The specific problem is classifying job titles into various industry codes. For example "McDonalds Employee" might get classified to ...
0
votes
1answer
19 views

Why dont we use 2d cnn filters for Nlp tasks?

CNNs are used in NLP for various tasks. But I cannot find a clear understanding of why do we only use 1d filters in these networks?
2
votes
1answer
23 views

Is there any way to calculate a relevance score between a title and the content of a text?

My question might sound a little bit stupid but I am trying to come up with a way to measure the relevance of the title of a text, let's say a piece of news headline, to the content. My idea would be ...
0
votes
2answers
99 views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...
1
vote
1answer
11 views

What are some of the available methods for handling multi-label classification for longer sequences of text

I am looking to solve a multi-class classification problem with long sequences of text with some rows having 1000's of tokens. Some of the state of the art methods such as BERT have a token limit and ...
0
votes
0answers
171 views

Keras: Vector regression: ValueError:can not squeeze dim[1]

I'm trying to create a NLP model which takes x_train_padded_2 (padded/tokenized text sequences) as input and try to approximate ...
1
vote
1answer
317 views

Overfitting with text classification using Transformers

I am trying to make a binary text classification model by using the encoder part of the transformer and then using its output to feed into an LSTM network. However, I am not able to achieve good ...
0
votes
1answer
13 views

Gender identification task on instance or user level?

I'm working on a task which is gender identification. Given a user account (e.g. Twitter account) with its documents (e.g. 100 tweets), the user should be classified as a male or a female. The ...
0
votes
0answers
13 views

Named Entity Recognition/Linking with a dictionary of names (e.g., move titles, book titles, company names, people names)

I have a lot of text messages on which I want to perform Named Entity Recognition and Named Entity Linking. For example, I want to detect all movie names and find their IMDB ID. To make things easier,...
0
votes
3answers
27 views

approach to classify text with natural language processing methods

I have a problem with regards to text classification/categorization. The task is bugging me for days already and as I am pretty new to AI and the field of natural language processing (NLP) I am just ...
0
votes
2answers
187 views

BERT classifier with Ktrain API is unable to predict new data

I have trained a classifier for sentiment analysis using BERT architecture. I am able to train the classifier and I am getting a validation accuracy of 87%. But whenever I feed in test data, or some ...
0
votes
0answers
15 views

Known datasets for long document analysis

Are there good baselines datasets or benchmarks for similarities/retrieval of long texts (extremely long documents, books, etc.)? Although significant both academically and applicational, I cannot get ...
0
votes
1answer
42 views

How to implement LSTM using Doc2Vec vectors to get representation?

Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...

1
2 3 4 5