Questions tagged [text]
Text is a type of data often used in data science projects involving natural language processing.
164
questions
0
votes
0
answers
19
views
Can this task for phrases be called lemmatization?
I want to 'lemmatize' phrases to dictionary entries. For instance, the following collocates can be standardized to the idiom in the aforementioned link ...
- 436
0
votes
0
answers
64
views
Creating variations of prompts for ChatGPT
I am developing a fine tune model to emulate a tech support chatbot based on my given information. I am struggling to create a large dataset (aiming for 1000 prompt/completion pairs), does anyone have ...
- 1
1
vote
1
answer
39
views
Dictionary-based text analysis- dealing with length
I am working on an analysis using a dictionary-based text-as-data approach. I have a dataset of texts (n=1200), and I am applying a dictionary of 50 words (I tokenize the text with each word being one ...
0
votes
0
answers
9
views
Get most useful sentences from one text and add to other
Supposed i have 2 text. The are share the same topic. I would like to get the most useful sentences from one text and add to other. By 'useful' we can assume any similarity suitable function.
What are ...
- 113
0
votes
0
answers
8
views
Identifying and removing repeated text from json files post inference
Good morning: We index and summarize medical records using a proprietary algorithm. Being medical records they repeat the same information almost on every single page, like facility name, medical ...
1
vote
1
answer
26
views
how to extract common aspects from text using deep learning?
Can you suggest me some papers to read about deep learning models that find patterns/similarities between different texts?
What I have is a set of reviews with the following categories for each review:...
0
votes
0
answers
5
views
How to check how similar or different two texts are based on their cluster membership?
I have generated clusters based on crowdsourced ratings of a set of comments (collected from two different platforms). Each comment was rated by three different annotators using a likert scale of 3 ...
- 1
1
vote
1
answer
12
views
Predictive value of short text fields
I am working on a classification model using one of the following three algorithms: RandomForestClassifier, a TensorFlow model and a LogisticRegression model.
The data set I am working with has a ...
- 13
0
votes
0
answers
20
views
How to generate question based on information given in question or by combining multiple questions?
I have a dataset of question and I want to generate the questions from the given text. For example, if I have a task like "John paid 1500$ for television set. Television set has 25% discount. ...
- 1
0
votes
0
answers
16
views
Predicting whether or not text is of a specified topic (topic defined by key words and phrases)
I was looking into binary classification methods for classifying whether or not a given text is related to a topic that is defined by given key words and phrases (e.g. the topic is meals and key ...
0
votes
0
answers
24
views
Unstructured Text Prediction
I have a question regarding a real-world problem on medical data. I have an input and output dataset which looks like this:
Input
Output
Patient is sick with fever
Prescribe Panadol and tylenol
...
- 1
0
votes
0
answers
14
views
Text analysis - "interpret" bag of words
I wanted to try to create a machine learning tarot card reader. Setting up a simulated draw of tarot cards and their associated meanings is straight-forward enough. However, the listed applications I ...
- 101
0
votes
1
answer
18
views
Using relative or absolute frequencies to estimate group differences in texts
My objective is to estimate differences between how five political parties use moral words in their tweets and speeches. To that end, I have a dictionary that I pass to each tweet text / audio ...
1
vote
1
answer
25
views
Next-word Generation in Tabular Dataset
I'll build next-word generation using Tensorflow to predict address mapping. But, I saw many tutorial, next-word generation use long-text narration for its training dataset. But, I have dataset ...
- 41
2
votes
2
answers
517
views
Suggestions for guided NLP online courses - Beginner 101
I would like to know from the data science community here for suggestions on nlp courses.
I am new to NLP area and would like to take up a course which covers from basic to advanced concepts such as ...
- 2,449
0
votes
0
answers
14
views
Article extraction from newspapers
Currently I'm working on a task that involves having a page of a news paper and putting bounding boxes around each individual article.The first approach I thought of was using visual features to ...
- 1
1
vote
1
answer
144
views
How to calculate Pointwise Mutual Information (PMI) when working with multiple ngrams
Pointwise Mutual Information or PMI for short is given as
Which is the same as:
Where BigramOccurrences is number of times bigram appears as feature, 1stWordOccurrences is number of times 1st word ...
0
votes
0
answers
18
views
Can I get un-normalized vectors from the TF USE model?
I'm using this Universal Sentence Encoder (USE) model to get embeddings of a set of texts, each text corresponding to a newspaper article. In order to build a Recommender System, I generate user ...
1
vote
1
answer
35
views
What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?
For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work
this is the logic behind google's did you mean algorithm - used for spell correction suggestion.
What ...
- 11
3
votes
2
answers
103
views
NLP text representation techniques that preserve word order in sentence?
I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
- 397
0
votes
0
answers
22
views
How to convert a string variable containing comments to a variable with integers to be used in neural networks?
I am working with data contains comment variable like imdb data.
...
1
vote
1
answer
151
views
How to down\up sample text?
I have data set of 5566 samples - one column is the text of the recipe description and the other is what tax class is it.
I wish to make a classifier that would classify receipts using ML only.
I have ...
- 23
1
vote
1
answer
128
views
Find best features for text classification with Countvectorizer Python
I'm facing a text classification problem where the algo is human-made but impacted by keywords. Hence I can't use any ML model but I certainly can take a data-science driven approach to find the best ...
- 113
1
vote
1
answer
19
views
Partially labelled open-class classification problem with heavy overlap
Let's say we have a corpus of text, including discussions about movies and about sports. Unsupervised clustering would typically cluster into the two topics of discussion. However, we are interested ...
1
vote
1
answer
172
views
Matching 2 keywords list using NLP
I have two lists and I want to identify which elements are common (same or similar in meaning or context) in the list. Which NLP algorithm we should use.
...
0
votes
2
answers
1k
views
How to set vocabulary size, padding length and embedding dimension in LSTM network?
Usually in a LSTM network, we have certain parameters that need to be set before the model can begin training. I am specifically talking about vocabulary size, padding length and embedding dimension. ...
- 1,321
0
votes
2
answers
161
views
Keywords extraction for business rule text classification
I would like to classify texts without using any ML model. My idea is to find a list of keywords that I would assign to each class. Then when I need to classify a new text, I can compare it with my ...
- 113
1
vote
1
answer
135
views
Why not rule-based semantic role labelling?
I have recently found some interest in automatic semantic role labelling. Most introductory texts (e.g. Jurafsky and Martin, 2008) present approaches based on supervised machine learning, often using ...
- 113
0
votes
0
answers
203
views
Text classification length
I have a set of text examples I need to learn as class A, and they are of varying lengths, say 10 sentences to 1 sentence long. I have to parse a document to find those strings of text that match one ...
- 115
0
votes
0
answers
99
views
NLP conversation data - Pre-processing steps
I have text data, so data that has been transcribed from conversations from employees to customers. So each call has a recording that has been transcribed to text.
I am looking to do some analytics ...
5
votes
1
answer
1k
views
How to evaluate the similarity of two columns containing strings?
I am new to text processing and stuck on a problem to identify the similarity of columns.
To detail the problem, consider we have two columns with string values:
...
- 203
0
votes
0
answers
27
views
Problem Direction - Text Data - Conversation Classification
The problem
I have a problem where I have text data that has been transcribed from a conversation. These conversations have been marked as a pass or fail in terms of compliance by a person, ie they ...
0
votes
0
answers
29
views
How do I work with noisy real world text data for text classification?
I have a topic classification model built upon Bert, when I deploy my model people input strings of a random nature like :
"aaaaaa" "aaa bbb" "ab ab ab"
and so on.
My ...
0
votes
1
answer
37
views
Working with three types of data: numeric (integer, floats), images, and text for prediction
So I have three types of data (in title) and am wondering how I can combine the data. The target is numeric (price).
My idea is to perform feature extraction on both the images and text, which would ...
1
vote
1
answer
80
views
Topic modelling on long documents: intra document clustering first
I have a collection (around 1000) of very noisy, similar documents, that are each very long (>10 pages - 600 paragraphs) with multiple subsections - I want to perform topic modelling across the ...
2
votes
1
answer
28
views
Classifying short strings of text with additional context
I have a list of short strings each identifying a city. Misspellings are very common. The example below shows some of these short strings, along with the correct city they're supposed to match.
...
- 165
2
votes
1
answer
2k
views
How to impute missing text data?
Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values?
If it ...
- 215
1
vote
1
answer
111
views
How to find the probability of a word to belong to a dataset of text
I have two text datasets, one with people that have a certain medical condition and another of random patients. I want to figure out what words are more likely to show up in the dataset with that ...
- 261
1
vote
1
answer
21
views
Performance measurement of an event extraction system
I have developed an event extraction system from text documents. It first clusters the data corpus and extracts answers for what, when and where questions. Final answers are determined by using a ...
0
votes
1
answer
820
views
ValueError: Layer model expects 2 input(s), but it received 3 input tensors using generator
I am trying to fit a model using generator function and I get the following error:
...
4
votes
1
answer
59
views
encoding of text data in NLP
I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
- 43
1
vote
1
answer
614
views
Cluster words into groups of similar meaning (synonyms)
How can words be clustered into groups of similar meaning (synonyms)?
I started with pre-trained word embeddings (e.g., Google News), which is great, but not perfect - a limitation arises because the ...
- 131
0
votes
0
answers
46
views
Effective way to find similarity between utterance(short text) and question(long text)
The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
0
votes
1
answer
293
views
Advantages of CNN vs. LSTM for sequence data like text or log-files
When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what ...
- 1
0
votes
1
answer
77
views
How to identify the feature that make the model misclassifed in text classification
Hi I am working on social media financial THAI text classification, the problem with this one is the confused classes, the misclassified prediction has a pattern that consistent as a pair.
and I want ...
- 1
0
votes
0
answers
26
views
text elaboration with neural network non natural language
I'm trying to understand if it's possible to elaborate with the neural network 2 input strings and predict a result string. I'm doing this job manually and it's really time consuming: basically i have ...
- 1
0
votes
1
answer
200
views
Word embedding for a single word
I want to tackle email similarity with a word embedding approach, not an expert in embedding text but it is possible to embed emails where similar emails have similar vectors? is that okay?. I know ...
- 111
0
votes
1
answer
154
views
How to remove irrelevant text data from a large dataset
I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed ...
- 113
3
votes
1
answer
733
views
Extract date/duration from text
The text and output to be extracted are similar to the following :
"Check it every two weeks" - two weeks
"Check it on day 1 and day 14" - day 1 and day 14
"day 19 and day ...
- 31
0
votes
0
answers
2k
views
Excel error may be caused by pandas writing or big data? advise needed
I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do:
open an xml file
extract some elements
create a pandas dataframe ...
- 1