Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,491
questions
0
votes
0
answers
63
views
How to create synthetic data with custom feature values?
Currently I create a synthetic dataset using the python sdv package and an existing dataset as below:
...
- 21
2
votes
0
answers
1k
views
How many samples in dataset are required to fine-tune BERT for binary classification?
I'm trying to fine-tune a BERT-based model for a binary classification task (data is in English). The dataset I'm working with is quite small (~500 samples, out of which 80% are currently used for ...
- 21
1
vote
0
answers
182
views
Grouped stratified train-val-test split for a multilabel dataset
So this is indeed nontrivial. I was wondering if there is a fast heuristic algorithm for performing grouped stratified dataset split on a multilabel dataset.
Stratification is usually performed to ...
- 23
1
vote
1
answer
33
views
How to select target words for Lexical Simplification dataset
I am trying to compile a Lexical Simplification dataset, which contains sentences, target words, and their simpler substitutes. I have already found some similar datasets in various languages, here ...
- 21
1
vote
2
answers
66
views
How to evaluate Natural Question-Answer Generation pairs?
I am trying to generate Natural Question-Answer for a specific domain. I am using a Large Language Model (LLM). I have only context to generate question-answers but don't have any ground truth. How to ...
- 415
0
votes
0
answers
213
views
Is it possible to turn the gmail inbox into a dataset for AI fine tuning?
My idea is to have a dataset of my gmail emails and replies. The purpose is to create a bot that can reply new emails based on all past correspondence in my inbox. How do i prepare such dataset from ...
- 1
0
votes
1
answer
41
views
Why does PowerBI take longer to load data than PowerQuery takes to load preview?
I have a stored procedure in SQL Server and when I open PowerQuery, I can get a preview of the whole table in a minute or two. But once I click 'Apply and Close', it will take AGES to load and finally ...
1
vote
2
answers
30
views
Tool for visualizing the evolution of a dataset
I wonder if anyone knows of a tool that allows to visualize how a dataset has evolved over time.
There are tools for tracking experiments (neptune, mlflow, etc.) but they are based on keeping track of ...
- 241
1
vote
1
answer
67
views
Input shape - How to feed metadata to a ML model?
I have data such as metadata: hospital layout, number of rooms, number of patients in a day etc.
and then I have data regarding the doctor’s check-ins. Which is more granular.
How do I feed this data ...
- 11
1
vote
1
answer
29
views
Where/how share annotated datasets (curated by humans)?
Once we have invested resources into annotating and reviewing some datasets (manually, by humans) how should we proceed to share them?
How can we make them findable and accessible? Is it possible to ...
- 111
0
votes
0
answers
12
views
Should I standardise my domain specific text prior to annotation/creation of a Q&A dataset?
I'm going to create my own question and answer dataset to fine-tune a BERT model, however before starting I am trying to understand if any standardisation of the text needs to be performed.
The data I ...
- 1
3
votes
1
answer
533
views
Incremental clustering algorithm
I am looking for an incremental clustering algorithm.
By incremental I mean an algorithm that builds clusters starting from an initial dataset and that is able to progressively ingest new items/...
- 131
0
votes
1
answer
14
views
Why do we use multiple recordings from the same set in a dataset?
In the following dataset:
https://physionet.org/content/mitdb/1.0.0/
Has been written the following text:
"the remaining 25 recordings were selected from the same set to include less common but ...
- 111
0
votes
0
answers
15
views
Search for fundus data obtained with a peek retina, or a D-EYE, or an iNview
I want to develop a machine learning model for eye disease detection via a smartphone.
For this I am looking for data. Especially images of the back of the eye obtained with a Peek retina, a D-eye or ...
0
votes
1
answer
25
views
How to carry metadata with data examples and labels in python?
Is there a pythonic way to carry around metadata that describes the training examples, such that it preserves (i.e., order) after shuffling and splitting (train/test)?
...
2
votes
2
answers
75
views
Difference in proportion between labelled sample and the population
I'm working on a project to predict bots from legit users from social medias. The data that I collected has about 5% of bots for 95% of legit users. The problem is as I labelled my data, I was more ...
- 222
0
votes
1
answer
608
views
How do you Specify which Columns Pandas reads in?
I have a huge dataset with ~470,000 rows and ~1400 columns. I only require 184 of the available columns. I'd like to limit the columns I read in to improve my dataset's loading times.
How do you ...
- 617
1
vote
2
answers
36
views
why it would be improper to compute and use test set means?
I have 2 questions regarding the whole subject of the data set in machine learning and I would be happy to receive an answer :)
1.Why it would be improper to compute and use test set means and ...
- 9
2
votes
1
answer
626
views
How do you Speed up the Calculation of a Correlation Matrix on a Large Dataset in Pandas?
I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' ...
- 617
0
votes
1
answer
318
views
How to assign a random number to multiple columns based on a condition in R?
I have a data frame for an experiment where participants had to answer 5 questions. The system I used for the experiment only reported the results as pass or fail. So basically a score of 1 for ...
1
vote
1
answer
48
views
Sentiment Analysis models trained on articles / alternative data
For a 6 class sentence classification task (emotion), I have a list of sentences where I retrieve the sentiment using a language model that was trained on Tweets (bertweet).
It works fine for ...
- 215
0
votes
1
answer
24
views
How do I use one hot encoding with 240 nominal variables and each with equal occurrence?
The method I saw that's generally used to deal with large # of nominal variables is to keep the most frequent variables and introduce a new "other" category. But that's not possible with my ...
- 1
1
vote
0
answers
42
views
Understanding Syntactic divergence
I am trying to read a paper https://arxiv.org/abs/2004.14444 Section 6.1 of the paper describes Syntactic divergence. I have confusion regarding the distribution graphs and split of the dataset.
If a ...
- 11
0
votes
0
answers
21
views
how to predict most frequent purchased item in e-commerce?
i have the following dataset:
witch its market transaction dataset and i need to predict the most frequent purchased items based on transaction history, for example: if sara bought milk and cookies ...
0
votes
1
answer
30
views
How to train the machine to read xray image in correct position?
I have trained a model to classify Xray of various body parts using CNN.
Now I want the model to read a Xray image in correct position even if input image is given in wrong position(rotated).
I think ...
- 1
1
vote
2
answers
50
views
Open source NLP annotation tool/library supports active learning
I am looking for an NLP annotation tool/library that supports active learning. I am looking for something that works in this scenario:
Annotating N samples.
Training a model on the annotated data.
...
- 252
0
votes
0
answers
227
views
Error message using OneHotEncode
I wrote this code trying to encode male and female gender in two different columns but keep getting an error message
code:
...
- 1
0
votes
1
answer
950
views
What can I do when my object detection model learns background images instead the objects?
I'm training a machine learning model using YOLOv5 from Ultralytics (arch: YOLOv5s6).
The task is to detect and identify laundry symbols.
For that, I've scraped and labeled 600 images from Google.
...
- 123
0
votes
1
answer
41
views
In between data partition and feature selection which one need to perform 1st
I need to perform a feature selection on my dataset. My dataset is an imbalanced dataset where the class of interest is the minority class. Therefore, recall and F2 measures are two important metrics ...
- 359
0
votes
1
answer
525
views
Using Python/Pandas to compare columns of unequal length
https://ibb.co/cyCzVgy
I would like to be able to compare the NEW_PRICE to the LINE_TOTAL_ALLOWED amounts based on the ...
- 1
0
votes
2
answers
52
views
How to manage large datasets (approx 95GB)
I was planning some data analysis on a dataset I'll be using for some projects. The dataset in question is ZINC20. Now, I don't need the whole thing so I was going to write some functions that would ...
- 1
0
votes
1
answer
47
views
Transform dataset to regression problem by sorting?
I have a raw unlabeled dataset, and I want to design a model to perform a regression. In my dataset, it does not make sense to give each observation a value, but it does make sense to sort them. Can I ...
1
vote
1
answer
33
views
Combine datasets of different domains to ehance generalizibility
so I try to implement an Emotion Classifier, which should detect several emotions from a text. There are several datasets for this (ISear, GoEmotions, etc.). However, a lot of them come from different ...
- 13
1
vote
0
answers
18
views
What is the best way for me to classify this audio data?
I have a set of audio data. I would like to classify each audio file based on a half-second of data from a give time period. The audio data is given as counts as a function of time $s(t)$. Right now ...
- 11
1
vote
0
answers
9
views
SEER dataset relation between attributes
I am working on SEER breast cancer dataset. It is an open source dataset. SEER dataset. I would like to know what is the relationship between attribute survival month and the class attribute Status?
...
- 359
0
votes
1
answer
38
views
Fraud Pattern Detection Dataset Creation
I have simulated a dataset for clicks on YouTube videos that records each click using dummy data. In the dataset, I collect information such as timestamp, ...
- 215
0
votes
0
answers
39
views
Is my image too low res for semantic segmentation task?
I am trying to solve a semantic segmentation task in the field of agriculture and I have some ortophoto drone images that have been acquired at the same height above the crops and in different ...
2
votes
1
answer
109
views
Training Neural Network using TensorFlow on Large Video Dataset for Human Activity Recognition
I am working on a Human Activity Recognition on videos using deep learning and want to train my Neural Network (ConvLSTM & LRCN) on my custom dataset. The issue that I am facing is that my dataset ...
- 21
1
vote
2
answers
216
views
How can I create new dataset maintaining the same trend from an existing dataset?
I am trying to create several new datasets based on an existing dataset. The new datasets should follow the trends (i.e., the correlation within the data) of that existing dataset.
As an example, I ...
- 131
0
votes
1
answer
29
views
How to aggregate qualitative results from a simulation
I am trying to look for methods to aggregate and find the average of qualitative data that are outputted from a simulation.
There are 20 qualitative measures, each divided unevenly into 4 cycles ...
- 111
1
vote
0
answers
45
views
supply chain analytics dataset?
I need to develop a model which enhances the supply chain and reduces the cost and wastage for perishable products like meat, seafood, dairy products and fruits and vegetables etc. Could someone ...
0
votes
0
answers
960
views
train_test_split error: Found input variables with inconsistent numbers of samples
I am trying to split a dataframe into training and test set and I am getting this error: "train_test_split error: Found input variables with inconsistent numbers of samples: [8819, 2795]".
...
- 11
1
vote
0
answers
11
views
Can I know if there's anyway to download Tableau report from server along with Title and Description in excel form?
I would like to download a Tableau report with the title footer and description written in Crosstab as it gets in PDF?
- 11
0
votes
2
answers
39
views
Difficulty loading data/running model on custom dataset derrived from DNA sequence data - TypeError when attempting to run model
I am a student who has some limited experience with keras, and for a new project recently decided to learn how to use pytorch to implement my models. I'm a beginner with both, so apologies in advance ...
0
votes
0
answers
924
views
How to find the target variable in the dataset?
I am working on the following dataset: https://drive.google.com/file/d/1Xzvw-z_glMcOYxJPJ4SckuKrVOulUuzO/view?usp=sharing
My goal is to predict whether the customer would check in or not. I am having ...
- 111
1
vote
1
answer
66
views
More representative data set OR higher model evaluation metrics?
A colleague and I are working on a churn model and reached an impasse:
Our data set is for a global product. We've been asked to look at the US market only.
When we subset the data to the US only, the ...
- 111
1
vote
1
answer
31
views
Cannot figure out how to make THIS particular data set long-form (theory, not even code)
I am a tableau developer, but I know Python, stats, and, in short, I think you all will be best able to solve my problem.
There is a universal filter on Facility. This means that any dataset/sheet ...
- 21
0
votes
0
answers
66
views
Why my neural network model is not able to learn a simple linear function?
I'm trying, using a neural network, to predict a simple relation $ f(x) = 1-x $.
I write my function this way: $f(x) = 1 - x^{+} + (-x)^{+} $ or in a more data-scientistic way: $f(x)= 1-Relu(x)+Relu(-...
0
votes
1
answer
674
views
Is it bad to have similar images in training and validation set?
I saw many persons on YouTube who are making videos, split them into frames and label each object for a custom dataset. Therefore there're many images of a single scene which are very similar to each ...
0
votes
1
answer
19
views
Distribution / number of examples
Imagine you want to predict if a picture is showing a cat or not. First you train your ML algorithm with examples of pictures of cats and dogs and it works. But then you want to train it to also ...
