Questions tagged [nlp]

Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Best way to build gensim WdmSimilarity for document data

I'm building an application that searches for queries in OCR data. My documents are numerous and have many pages. I'm using Wmd Similarity to query my data with gensim ...
0
votes
0answers
14 views

Does batch size matter in inferencing speed?

I am reading the paper "Are Sixteen Heads Really Better than One?" and in section 4.3 it states that the inference speed varies with batch size. How does batch size affect inference speed ...
-1
votes
0answers
5 views

How do I use multiple workspaces(skills) in IBM WATSON Assistant? [closed]

I have tried the methods available on the internet but no sheer luck. I also followed the articles available on Medium.
0
votes
1answer
10 views

How can I preprocess text to feed into a SVM?

I am using an IMDB dataset which contains reviews of the movies in the column text and the rating 0 or 1 in the column label. I am preprocessing the text using Tfidf using sklearn. The code for the ...
0
votes
0answers
12 views

to generate consistent encoding for words in Keras using tf.keras.preprocessing.text.one_hot

I am using keras(tensorflow) to convert text into encodings using tensorflow.keras.preprocessing.text.one_hot I have used it for training dataset as below ...
-1
votes
0answers
6 views

how to efine semantic similarity measure between job titles?

I am wondering how I can define a similarity measure for similar job titles? It has to be semantic For example I have different type of specialist and I want to define a similarity measure between ...
0
votes
0answers
12 views

Extracting structured data from semi structured data

I want to use machine learning and NLP to convert semi-structured data in text files to structured data by predicting the patterns in the files and splitting the fields for example if I have a text ...
1
vote
0answers
18 views

About Natural Question (NQ) benchmark in NLP [closed]

I recently learned that there is a benchmark called NQ. https://ai.google.com/research/NaturalQuestions/visualization Unlike other QA benchmarks which relevant document is povided with query, it has ...
0
votes
0answers
16 views

Generate Knowledge graph from text paragraph as input using python

I want to plot a knowledge graph from text paragraph as input in python, So wanted to know is there any transfer learning model, machine learning method or python library which helps in generating ...
1
vote
0answers
10 views

Is it domain adaptation?

Let's say I trained a classifier to assign a shop department to a product, e.g.: ALGIDA Cow milk->Diary. It did it on a domain of official product names. When I applied pre-trained classifier to ...
0
votes
0answers
26 views

Classifies the place of birth that belongs to the person at NER

I want to classify the place of birth and date of birth for each person detected by the NER results. For example, I have a sentence like this: This paper represents the fact that Jaden Smith was born ...
0
votes
0answers
24 views

Looking for more recent dataset for document classfication

I am trying to develop an NLP - CNN algorithm to detect documents with sensitive information such as passport, license and distinguish them from other documents like resume, email, form or ...
0
votes
0answers
17 views

Text Classification using Language models like Bert, XLnet for 10 million data point [closed]

I am trying to train a Bert/ XLnet for the Classification task. The problem lies in the fact that training and inferencing speed is extremely slow. For a dataset size of a million, it takes more than ...
-1
votes
0answers
20 views

NLP to catch information about trades ASAP [closed]

I want to build a model to scan tweets/news etc and alert if it's about trades that involves new york yankees. It may be about the Yankees acquiring a player from Mets or releasing players, etc. I do ...
0
votes
0answers
12 views

Change wav format [closed]

I am working with wav files that are not comptaible with the wav library and I keep getting the error: Error: unknown format: 6 Does anyone know how to get over ...
1
vote
1answer
21 views

What is the difference between Okapi bm25 and NMSLIB?

I was trying to make a search system and then I got to know about Okapi bm25 which is a ranking function like tf-idf. You can make an index of your corpus and later ...
0
votes
1answer
11 views

How to reduce the GPU consumption size while using Elmo Model?

I am performing an NLP task using Elmo model. Whenever I load the Elmo model, it occupies the 15 GB of my GPU memory. How can I reduce it ? Below is my code ...
-1
votes
0answers
18 views

Speaker Diarization splitting wav file based on speaker [closed]

i am working in with a Speaker Diarization task and i splited my wav file using spectral clustring and i have now a list of tuples with values in order (speaker_label, start_time, end_time) and the ...
3
votes
3answers
137 views

Binary classification and numerical labels

I am trying to create a sentiment analysis model using a dataset that have ~50000 positive tweets that i labeled as 1, ~50000 negative tweets that i have labeled as 0. Also i have acquired ~10000 ...
0
votes
0answers
11 views

How to train a text classifier for product search query to determine category

I am trying to train a product search query (e-commerce) classifier for deducing probable product categories from search query with a dataset of 700k queries with probable categories labelled I tried ...
1
vote
0answers
8 views

How to add words to english model word list in Julius Speech Recognition Engine?

I want to add some English words to model but how can I achieve this ? https://github.com/julius-speech/julius
1
vote
1answer
17 views

NER with LSTM - How to recognize person names that are not part of the vocabulary?

I am learning Named Entity Recognition and going through posts similar to this one: Named-Entity Recognition (NER) using Keras Bidirectional LSTM So the sentences are fed into the model as a sequence ...
1
vote
1answer
26 views

Cross between an edit distance algorithm and a phonetic algorithm

My aim is to find an edit distance algorithm which penalises transformations differently depending on the phonetic context. Take the Levenshtein algorithm, for example; it penalises the same operation ...
0
votes
1answer
33 views

Are all 110 million parameter in bert are trainable

I am trying to understand are all these 110 million parameters trainable of bert uncased model. Is there any non trainable parameters in this image below? By trainable I understand they are ...
0
votes
0answers
24 views

How does Google's Universal Sentence Encoder deal with out-of-vocabulary terms?

It seems to output embeddings even for random jibberish, and the similarity is even high for this particular pair of jibberish. ...
0
votes
1answer
31 views

Backpropagation of a transformer

when a transformer model is trained there is linear layer in the end of decoder which i understand is a fully connected neural network. During training of a transformer model when a loss is obtained ...
6
votes
2answers
272 views

Python stemmer for Georgian

I am currently working with Georgian texts processing. Does anybody know any stemmers/lemmatizers (or other NLP tools) for Georgian that I could use with Python. Thanks in advance!
0
votes
1answer
17 views

How to stem plural words properly?

I'm looking for a way to avoid removing ending s when s isn't a suffix. In order to do that, I first check if a word exists in ...
0
votes
0answers
16 views

Topic modelling with many synonyms - how to extract 'latent themes'

Here's my corpus ...
0
votes
0answers
17 views

Clustering together words that appear together while down weighting words that appear too often

I was wondering if I could get some help finding a good model for the problem I have. I have a data set where each observation is a set of words that go together. So for example, it could be: ...
1
vote
0answers
20 views

Word2Vec vs. Doc2Vec Word Vectors

I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access. My ...
1
vote
1answer
32 views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
-2
votes
1answer
47 views

Transformers understanding

i have i a big trouble. I don't understand transformers. I understand embedding, rnn's, GAN's, even Attention. But i don't understand transformers. Approximately 2 months ago i decided to avoid usage ...
0
votes
1answer
11 views

Parse documents to obtain subjective sentiment

I'm working on a project which deals with MRC (Machine Reading Comprehension). I would like a machine to read an article and give me the sentiments based on a provided token. For Instance given the ...
4
votes
1answer
81 views

How to preprocess with NLP a big dataset for text classification

TL;DR I've never done nlp before and I feel like I'm not doing it in the good way. I'd like to know if I'm really doing things in a bad way since the beginning or ...
1
vote
1answer
24 views

“Rare words” on vocabulary

I am trying to create a sentiment analysis model and I have a question. After I preprocessed my tweets and created my vocabulary I've noticed that I have words that appear less than 5 times in my ...
-1
votes
0answers
12 views

How to remove irrelevant text data from a large dataset

As title says, I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data ...
-1
votes
1answer
24 views

Printing the tweets that were incorrectly predicted after applying a machine learning classifier

I applied the random forest classifier to my csv file to classify the tweets as spam or not spam and after an accuracy of 93%, when I printed the confusion matrix I got [[1068 105] [ 65 1262]]. Now ...
0
votes
1answer
51 views

What is the difference between CountVectorizer() and Tokenizer() or are they the same?

from sklearn.feature_extraction.text import CountVectorizer from keras.preprocessing.text import Tokenizer I am going through some NLP tutorials and realised that ...
0
votes
0answers
24 views

Use Categorical features in BERT model

I am trying to fine-tune BERT-base model for binary text classification using multiple features. 3 text features, 4 categorical features. Text features having more than 500 tokens length, and four ...
0
votes
1answer
12 views

Best approach to create a text classification model with two inputs?

I'm looking to train a model with two text inputs (sentences) and a binary classification. Essentially, for 2 given sentences, are they paraphrases or not. I want to use the Microsoft research ...
0
votes
0answers
11 views

Compact model for on-device next word prediction

I'm an iOS developer with no production ML experience besides some pet projects. The task is to create a model to predict the next word in English for a custom keyboard. I'd actually prefer to just ...
0
votes
1answer
15 views

Transformer architecture question

I am hand-coding a transformer (https://arxiv.org/pdf/1706.03762.pdf) based primarily on the instructions I found at this blog: http://jalammar.github.io/illustrated-transformer/. The first attention ...
1
vote
1answer
9 views

Extracting events with attributes from unstructured text

I am scraping websites of organisations (mostly retailers) and I want to use NLP to extract information from the websites’ unstructured text. The first thing I want to do is to identify covid-related ...
-1
votes
0answers
29 views

All classifiers seems to give very good evaluation metrics results

I'm currently doing a topic categorization task with the help of the sklearn library on news articles by using the BBC Datasets (http://mlg.ucd.ie/datasets/bbc.html). I chose 5 classifiers namely SVM, ...
1
vote
0answers
16 views

NLP date detection/extraction [closed]

I'd like to detect Dates in Texts. This should include strings like 'last friday', 'next week' and so on. Unfortunately I couldn't find anything useful in the literature. Does anyone know a good ...
1
vote
1answer
26 views

what's the motivation behind BERT masking 2 words in a sentence?

bert and the more recent t5 ablation study, agree that using a denoising objective always results in better downstream task performance compared to a language model where denoising == masked-lm == ...
2
votes
1answer
47 views

How to evaluate the quality of speech-to-text data without access to the true labels?

I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription ...
1
vote
0answers
23 views

Pytorch CTC Loss Unexpected Behaviour

I have used the following code to test the behaviour of CTC loss. ...
1
vote
1answer
21 views

what is the training phase in N-gram model?

Following is my understanding of N gram model used in text prediction case : Given a sentence say, " I love my " (say N = 1 /bigram), using N gram and say 4 possible candidates ( country, ...

1
2 3 4 5
33