Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,463
questions
0
votes
1
answer
12
views
CWRU Bearing fault
I am new to ML. I have been asked to use a pre-trained GRU model for detecting a bearing fault in CWRU. is pre-trained model another name for transfer learned model?
0
votes
0
answers
20
views
Should I use Boruta feature selection if I'm comparing two datasets?
I'm a beginner to machine learning and AI so I really could you some guidance on this.
I am currently comparing two different datasets with different classification algorithms. One of the datasets has ...
- 13
0
votes
0
answers
21
views
Finding a relation between three variables
I am new to data mining and have learnt about association rules mining, classification analysis, cluster analysis and outlier analysis. So, to find relationship between three variables, regression ...
- 1
3
votes
0
answers
51
views
Public data set with exciting data
I'm going to give a talk about a tool for viewing tabular datasets. I'll first give this talk at a meetup, then hopefully at a series of conferences over the next few years.
I want to demonstrate to ...
- 255
0
votes
0
answers
65
views
What is the difference between SMOTE before PCA and after PCA
We all know that PCA (Principal Component Analysis) is a popular statistical tool to reduce the dimensionality in a dataset. SMOTE (Synthetic Minority Over-sampling Technique) allows you to generate ...
1
vote
2
answers
42
views
how to build an images dataset efficiently
I have several image datasets and I am going to combine all of them into 1 giant dataset I decided to load the images which are in different formats (SVG, PIL images, etc ) and save them as png but ...
- 135
0
votes
0
answers
17
views
How to load time series data set, from .csv into keras 3D tensor
I would like to load, from CSV, where data are listed row by row, to load into Keras 3D tensor for LTMS. I've got error, that it requires 3D data, but I've only got 2D from CSV.
Thank you in advance.
0
votes
2
answers
86
views
When is the right moment to split the dataset?
I would like to ask a question about the dataset splitting.
If I have a dataset can I perform preprocessing (imputation, scaling, ecc.) on the entire dataset and then splitting it into train and ...
0
votes
1
answer
36
views
I'm Looking for dataset with Name, Father's Name, Mother's name, Age, Gender, Ethnicity, ~Birthplace [closed]
I am new to Data Science and I don't know where to find datasets, I Google searched, Asked ChatGPT and friends, but couldn't find what I was looking for.
My project is to generate new names based on ...
- 1
0
votes
0
answers
19
views
What is semi-ground truth?
What is semi-ground truth and how it is different from ground truth? Can you explain with an example?
- 1
0
votes
0
answers
14
views
What approach can be take to design a food menu intelligently?
I am given following problem, for which I am seeking for advice.
Problem Statement
Suppose you are given a catalogue of dishes with their cost and calories. Can you design an algorithm/model which can ...
- 207
0
votes
0
answers
35
views
Is this overfitting? (generative model)
I am working with a generative method, and the network seems to perform well on training data and slightly less well on test data, but the generated data is somehow significantly worse than either of ...
- 31
0
votes
1
answer
48
views
Alternatives to Toronto Book Corpus
As the toronto book corpus is no longer available (or rather, only in lowercase), I am looking for an alternative dataset of comparable language variety, quality, and size.
Any suggestions? The ...
0
votes
2
answers
338
views
How to do train-test split for multi-class classification
I am performing multi-class classification problem of different concentrations of Acetaminophen in a specified dataset. My data is in the form of images and I am using CNN. I have compiled all the ...
- 3
0
votes
0
answers
12
views
Speaker Recognition using Deep Learning
I am trying to develop a speaker recognition model using deep learning. I want to use the VoxCeleb2 audio dataset but I cannot find it. The one on their website has YouTube links, but I would rather ...
- 1
0
votes
0
answers
32
views
Why are all PyTorch dataloader proccesses in S state (interruptible sleep) except one?
Short version: I'm using PyTorch dataloading library to load data in parallel for training a deep learning model. When I look at the CPU usage with htop, I see a ...
- 377
1
vote
0
answers
21
views
How much data and computation power do I need to train a machine translation model using Transformer architecture?
I am working right now on creating a dataset to use in creating a machine translation model to translate between two dialects. I have two questions that I am trying to find an answer for:
How much ...
- 11
1
vote
1
answer
24
views
Should I drop duplicates over features but no target
I'm in a debate with someone about a problem where there are duplicates over features (i.e. $ X_1 = X_2 $ but $ Y_1 != Y_2 $).
My point of view is that we should keep those datas, as they can be ...
- 311
0
votes
1
answer
20
views
How do I handle "Greater than X" in a field of integers?
I've been tasked with cleaning a dataset with a "Drive Time" column that lists times taken to drive to a specific location in whole minutes. The values range from 3 to 180 minutes but there ...
- 1
0
votes
0
answers
9
views
tensorflow dataset 'plant_leaves' cannot be imported to google Colab : DownloadRetryLimitReachedError
I'm trying to upload a tensorflow dataset called plant_leaves but it won't work and it shows a runtime error :
0
votes
0
answers
14
views
How to combine multiple similar columns in a dataset derived of multiple sources?
I am trying to combine data from multiple sources. One issue I face is that the same columns are named differently. For example, the following refer to the same columns:
SampleID, sample id, sampleID,...
- 101
0
votes
0
answers
14
views
How to deal with degeneracies in data
I am working with a dataset where I want to predict some features (let's call them G) from some other features (we'll call them W). The problem is that there is no one-to-one mapping from W to G. In ...
- 31
0
votes
0
answers
20
views
How to scale a vector into a smaller vector?
I have a vector A of 6 integers [1000,900,800,700,600,500] and each number corresponds to another vector B [0,1,2,3,4,5].
How can I get a vector A_new of 5 integers if I scale the vector B to [0,0.25,...
- 11
0
votes
0
answers
13
views
Using subgroup discovery for a very small subset of a huge dataset?
I want to use subgroup discovery to find some interesting patterns in a small subset of my dataset. The dataset contains about 100.000 records, the subset I want to look at only 100 records. Does it ...
- 1
4
votes
2
answers
55
views
Merging data approach in Data Science projects
This is more of an infrastructural question about data science. How would you manage data merging in your GitHub repository?
As an example, as a data scientist I might be working on my branch and ...
0
votes
0
answers
14
views
How to create a synthetic dataset for object detection with GTA-V
I am planning to do a project of object detection in the off-road environment. There is not much labeled data available for this.
I was thinking about creating my own dataset with GTA-V but I don't ...
0
votes
0
answers
9
views
Set labels in R summary table
I'm creating a function in R for plotting and calculate an ANOVA model (because I have to repeat this operation a lot of time).
I wrote this function:
...
- 1
0
votes
0
answers
10
views
Is there data on the number of residential units in each building?
I am totally new to conducting data science, so I don't know if this is the place to ask, but I am trying to find data on the number of residential units in each building in the USA for a project I am ...
- 101
0
votes
1
answer
14
views
Total Retention Rate Calculated from Categories
I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working.
I am comparing the last 14 days (let's call it Period 1) to ...
0
votes
0
answers
24
views
Validation accuracy decreases when I use ImageDataGenerator instead of manually creating labels
I was getting 85 percent validation accuracy before using ImageDataGenerator, now validation accuracy for binary classification decreases even to 14 percent. and never passes 60 percent. I couldn't ...
- 1
1
vote
0
answers
28
views
How can I detect errors/irregularity in a dataset
I've been working on a pathfinding project using topographic data for my software development course, even though I've never really had calculus lessons.
The algorithm works fine by itself, however, ...
- 11
1
vote
1
answer
30
views
what labeling format has negative Bbox values in labels?
I have a labeled dataset for object detection
few thousands of images with annotation on csv file
the csv contains these columns
image_path, class, xmax, xmin, ymax, ymin
looks like Pascal voc format ...
- 170
0
votes
0
answers
119
views
Where can I get REDD dataset if http://redd.csail.mit.edu/ is not working?
Reference Energy Disaggregation Data Set (REDD) is used for research in Non-intrusive Load Monitoring (NILM). Every paper and blogs points towards http://redd.csail.mit.edu/ as data downling link. ...
- 103
1
vote
2
answers
83
views
How to decide the most suitable technique to handle Class Imbalance
For handling an imbalanced dataset, we have a variety of techniques like adjusting class weights, oversampling, undersampling, SMOTE and its different variations (RCSMOTE, GSMOTE, DBSMOTE).
My ...
- 173
1
vote
1
answer
42
views
If I'm comparing performance between two different datasets should sample and class size be uniform?
If I'm comparing performance between classification models on two different datasets should the number of samples per class, the number of classes, and features per sample be the relatively the same ...
- 13
0
votes
0
answers
15
views
How to reference an intermediary object in scala?
I am using spark/scala to do some transformations on a spark dataframe. I have the following code:
...
- 11
0
votes
0
answers
29
views
How to tokenize custom dataset?
I have generated a dataset of quadratic equations and I want to train T5 model to solve them.
Dataset looks like this:
Could somebody please give a hint on how to tokenize this correctly? What to do ...
0
votes
0
answers
16
views
Is there any datset which has human judgement scores between 2 english senctences only?
I am trying to propose an evaluation metric in NLP and I need to compare it with other baseline metric such as Bleu scores.
I need a dataset which has 2 english sentences (say same image being ...
0
votes
0
answers
29
views
How to create synthetic data with custom feature values?
Currently I create a synthetic dataset using the python sdv package and an existing dataset as below:
...
- 21
2
votes
0
answers
509
views
How many samples in dataset are required to fine-tune BERT for binary classification?
I'm trying to fine-tune a BERT-based model for a binary classification task (data is in English). The dataset I'm working with is quite small (~500 samples, out of which 80% are currently used for ...
- 21
1
vote
0
answers
116
views
Grouped stratified train-val-test split for a multilabel dataset
So this is indeed nontrivial. I was wondering if there is a fast heuristic algorithm for performing grouped stratified dataset split on a multilabel dataset.
Stratification is usually performed to ...
- 23
1
vote
1
answer
31
views
How to select target words for Lexical Simplification dataset
I am trying to compile a Lexical Simplification dataset, which contains sentences, target words, and their simpler substitutes. I have already found some similar datasets in various languages, here ...
- 21
1
vote
2
answers
47
views
How to evaluate Natural Question-Answer Generation pairs?
I am trying to generate Natural Question-Answer for a specific domain. I am using a Large Language Model (LLM). I have only context to generate question-answers but don't have any ground truth. How to ...
- 415
0
votes
0
answers
114
views
Is it possible to turn the gmail inbox into a dataset for AI fine tuning?
My idea is to have a dataset of my gmail emails and replies. The purpose is to create a bot that can reply new emails based on all past correspondence in my inbox. How do i prepare such dataset from ...
- 1
0
votes
1
answer
23
views
Why does PowerBI take longer to load data than PowerQuery takes to load preview?
I have a stored procedure in SQL Server and when I open PowerQuery, I can get a preview of the whole table in a minute or two. But once I click 'Apply and Close', it will take AGES to load and finally ...
1
vote
2
answers
28
views
Tool for visualizing the evolution of a dataset
I wonder if anyone knows of a tool that allows to visualize how a dataset has evolved over time.
There are tools for tracking experiments (neptune, mlflow, etc.) but they are based on keeping track of ...
- 241
1
vote
1
answer
54
views
Input shape - How to feed metadata to a ML model?
I have data such as metadata: hospital layout, number of rooms, number of patients in a day etc.
and then I have data regarding the doctor’s check-ins. Which is more granular.
How do I feed this data ...
- 11
1
vote
1
answer
29
views
Where/how share annotated datasets (curated by humans)?
Once we have invested resources into annotating and reviewing some datasets (manually, by humans) how should we proceed to share them?
How can we make them findable and accessible? Is it possible to ...
- 111
0
votes
0
answers
12
views
Should I standardise my domain specific text prior to annotation/creation of a Q&A dataset?
I'm going to create my own question and answer dataset to fine-tune a BERT model, however before starting I am trying to understand if any standardisation of the text needs to be performed.
The data I ...
- 1
2
votes
1
answer
171
views
Incremental clustering algorithm
I am looking for an incremental clustering algorithm.
By incremental I mean an algorithm that builds clusters starting from an initial dataset and that is able to progressively ingest new items/...
- 121