Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,493
questions
0
votes
0
answers
15
views
How to estimate this variable in an MILP formulation
This is my first question being asked here. I've thought about different methods to do it, but to no avail. I want to estimate a variable that is either 0 or a positive number. Then I want to use this ...
0
votes
0
answers
9
views
Implementing a dataset to Computer Vision Article
I want to implement the PIE dataset in the AgentFormer arch.
AgentFormer uses ETH and nuScene datasets. I successfully run these datasets on this arch. However, I couldn't take a good way with the PIE ...
- 1
1
vote
1
answer
65
views
Do you have to use clustering with SciKit-Learn's Mutual Information metric?
I'd like to calculate the mutual information between two datasets, but I'd prefer not to cluster them first.
I'm thinking of using SciKit-Learn's mutual_info_score ...
- 617
1
vote
1
answer
45
views
Making a netcdf data using xarray
I am very very new to the world of data science as I only started using it in my new job so I would really appreciate help from the community experts (maybe also in simple words :)).
I am trying to ...
- 125
0
votes
0
answers
7
views
Is there any way to aquire image and video URLs from a large amount of facebook ads for research?
I have aquired a lot of URLs to some political ads from the Meta ad library API. As far as i can tell, there is no way to automate the aquisation of the image and video URLs from these ads. Is this ...
4
votes
3
answers
379
views
What techniques are used to analyze data drift?
I've created a model that has recently started suffering from drift.
I believe the drift is due to changes in the dataset but I don't know how to show that quantitatively.
What techniques are ...
- 617
0
votes
0
answers
59
views
How to combine two datasets to create training data
I am working on a problem where I have say dataset 1 (D1) and dataset 2 (D2), now D1 has three features F1, F2, F3, and D2 has four features F1, F2, F3, and F4. The number of samples in D1 is 7000 and ...
0
votes
0
answers
9
views
What sources exist for getting access to public medical data, like incidences of a certain disease?
Is there any way to access data about the spread of a disease? Does the medical system make public any information about what kinds of medical issues they are seeing? Is it legal for them to store ...
- 131
0
votes
0
answers
34
views
Benchmark dataset with both images and tabular data
In the medical sector, there are situations in which an image dataset is associated with a tabular dataset containing different features but the same labels as the image dataset. For example, suppose ...
- 229
1
vote
2
answers
29
views
How can i implement an confucion matrix?
im trying to do a research but i need to make a confusion matrix how can i do that on this model?
https://www.kaggle.com/code/stpeteishii/race-classify-densenet201
Sorry im so so new to everything.
- 11
0
votes
1
answer
47
views
using a feature that is only available during training
I'm working on a project that aims to classify JIRA issues into their relevant owner group.
An issue has the following text features:
Summary
Description
Comments
all of which are text based.
During ...
- 199
0
votes
1
answer
40
views
How to Manipulate a Dataset
I have a raw dataset of Images I got from Kaggle, It has been classified already, but I want to randomly split the information in a 80:20 ratio between train and test, problem is since all information ...
0
votes
1
answer
36
views
Build a dataframe with multiple CSV
I am collecting metrics on 6 REST services from a microservices architecture. For each instant collected, I extract two CSV from each service. One CSV contains three latency metrics (99th quantile, ...
- 101
0
votes
1
answer
26
views
Survey on image retrieval datasets
I am on a survey about image retrieval datasets. I have found some, such as:
NUS-WIDE
Oxford5k
Oxford105k
Paris6k
MSCOCO
I have been way too confused about the detection metrics and the metrics they ...
- 3
0
votes
1
answer
16
views
CWRU Bearing fault
I am new to ML. I have been asked to use a pre-trained GRU model for detecting a bearing fault in CWRU. is pre-trained model another name for transfer learned model?
0
votes
0
answers
42
views
Should I use Boruta feature selection if I'm comparing two datasets?
I'm a beginner to machine learning and AI so I really could you some guidance on this.
I am currently comparing two different datasets with different classification algorithms. One of the datasets has ...
- 13
0
votes
0
answers
30
views
Finding a relation between three variables
I am new to data mining and have learnt about association rules mining, classification analysis, cluster analysis and outlier analysis. So, to find relationship between three variables, regression ...
- 1
3
votes
0
answers
51
views
Public data set with exciting data
I'm going to give a talk about a tool for viewing tabular datasets. I'll first give this talk at a meetup, then hopefully at a series of conferences over the next few years.
I want to demonstrate to ...
- 255
0
votes
0
answers
147
views
What is the difference between SMOTE before PCA and after PCA
We all know that PCA (Principal Component Analysis) is a popular statistical tool to reduce the dimensionality in a dataset. SMOTE (Synthetic Minority Over-sampling Technique) allows you to generate ...
1
vote
2
answers
51
views
how to build an images dataset efficiently
I have several image datasets and I am going to combine all of them into 1 giant dataset I decided to load the images which are in different formats (SVG, PIL images, etc ) and save them as png but ...
- 137
0
votes
0
answers
27
views
How to load time series data set, from .csv into keras 3D tensor
I would like to load, from CSV, where data are listed row by row, to load into Keras 3D tensor for LTMS. I've got error, that it requires 3D data, but I've only got 2D from CSV.
Thank you in advance.
0
votes
2
answers
318
views
When is the right moment to split the dataset?
I would like to ask a question about the dataset splitting.
If I have a dataset can I perform preprocessing (imputation, scaling, ecc.) on the entire dataset and then splitting it into train and ...
0
votes
1
answer
109
views
I'm Looking for dataset with Name, Father's Name, Mother's name, Age, Gender, Ethnicity, ~Birthplace [closed]
I am new to Data Science and I don't know where to find datasets, I Google searched, Asked ChatGPT and friends, but couldn't find what I was looking for.
My project is to generate new names based on ...
- 1
0
votes
0
answers
28
views
What is semi-ground truth?
What is semi-ground truth and how it is different from ground truth? Can you explain with an example?
- 1
0
votes
0
answers
14
views
What approach can be take to design a food menu intelligently?
I am given following problem, for which I am seeking for advice.
Problem Statement
Suppose you are given a catalogue of dishes with their cost and calories. Can you design an algorithm/model which can ...
- 207
0
votes
0
answers
38
views
Is this overfitting? (generative model)
I am working with a generative method, and the network seems to perform well on training data and slightly less well on test data, but the generated data is somehow significantly worse than either of ...
- 31
0
votes
1
answer
113
views
Alternatives to Toronto Book Corpus
As the toronto book corpus is no longer available (or rather, only in lowercase), I am looking for an alternative dataset of comparable language variety, quality, and size.
Any suggestions? The ...
2
votes
2
answers
1k
views
How to do train-test split for multi-class classification
I am performing multi-class classification problem of different concentrations of Acetaminophen in a specified dataset. My data is in the form of images and I am using CNN. I have compiled all the ...
- 23
1
vote
0
answers
15
views
Speaker Recognition using Deep Learning
I am trying to develop a speaker recognition model using deep learning. I want to use the VoxCeleb2 audio dataset but I cannot find it. The one on their website has YouTube links, but I would rather ...
- 11
0
votes
0
answers
41
views
Why are all PyTorch dataloader proccesses in S state (interruptible sleep) except one?
Short version: I'm using PyTorch dataloading library to load data in parallel for training a deep learning model. When I look at the CPU usage with htop, I see a ...
- 377
2
votes
0
answers
46
views
How much data and computation power do I need to train a machine translation model using Transformer architecture?
I am working right now on creating a dataset to use in creating a machine translation model to translate between two dialects. I have two questions that I am trying to find an answer for:
How much ...
- 21
1
vote
1
answer
29
views
Should I drop duplicates over features but no target
I'm in a debate with someone about a problem where there are duplicates over features (i.e. $ X_1 = X_2 $ but $ Y_1 != Y_2 $).
My point of view is that we should keep those datas, as they can be ...
- 323
0
votes
1
answer
20
views
How do I handle "Greater than X" in a field of integers?
I've been tasked with cleaning a dataset with a "Drive Time" column that lists times taken to drive to a specific location in whole minutes. The values range from 3 to 180 minutes but there ...
- 1
0
votes
0
answers
17
views
tensorflow dataset 'plant_leaves' cannot be imported to google Colab : DownloadRetryLimitReachedError
I'm trying to upload a tensorflow dataset called plant_leaves but it won't work and it shows a runtime error :
- 1
0
votes
0
answers
43
views
How to combine multiple similar columns in a dataset derived of multiple sources?
I am trying to combine data from multiple sources. One issue I face is that the same columns are named differently. For example, the following refer to the same columns:
SampleID, sample id, sampleID,...
- 101
0
votes
0
answers
19
views
How to deal with degeneracies in data
I am working with a dataset where I want to predict some features (let's call them G) from some other features (we'll call them W). The problem is that there is no one-to-one mapping from W to G. In ...
- 31
0
votes
0
answers
41
views
How to scale a vector into a smaller vector?
I have a vector A of 6 integers [1000,900,800,700,600,500] and each number corresponds to another vector B [0,1,2,3,4,5].
How can I get a vector A_new of 5 integers if I scale the vector B to [0,0.25,...
- 11
0
votes
0
answers
14
views
Using subgroup discovery for a very small subset of a huge dataset?
I want to use subgroup discovery to find some interesting patterns in a small subset of my dataset. The dataset contains about 100.000 records, the subset I want to look at only 100 records. Does it ...
- 1
4
votes
2
answers
57
views
Merging data approach in Data Science projects
This is more of an infrastructural question about data science. How would you manage data merging in your GitHub repository?
As an example, as a data scientist I might be working on my branch and ...
0
votes
0
answers
36
views
How to create a synthetic dataset for object detection with GTA-V
I am planning to do a project of object detection in the off-road environment. There is not much labeled data available for this.
I was thinking about creating my own dataset with GTA-V but I don't ...
0
votes
0
answers
18
views
Set labels in R summary table
I'm creating a function in R for plotting and calculate an ANOVA model (because I have to repeat this operation a lot of time).
I wrote this function:
...
- 1
0
votes
0
answers
12
views
Is there data on the number of residential units in each building?
I am totally new to conducting data science, so I don't know if this is the place to ask, but I am trying to find data on the number of residential units in each building in the USA for a project I am ...
- 101
0
votes
1
answer
17
views
Total Retention Rate Calculated from Categories
I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working.
I am comparing the last 14 days (let's call it Period 1) to ...
0
votes
0
answers
41
views
Validation accuracy decreases when I use ImageDataGenerator instead of manually creating labels
I was getting 85 percent validation accuracy before using ImageDataGenerator, now validation accuracy for binary classification decreases even to 14 percent. and never passes 60 percent. I couldn't ...
- 1
1
vote
0
answers
54
views
How can I detect errors/irregularity in a dataset
I've been working on a pathfinding project using topographic data for my software development course, even though I've never really had calculus lessons.
The algorithm works fine by itself, however, ...
- 11
1
vote
1
answer
58
views
what labeling format has negative Bbox values in labels?
I have a labeled dataset for object detection
few thousands of images with annotation on csv file
the csv contains these columns
image_path, class, xmax, xmin, ymax, ymin
looks like Pascal voc format ...
- 185
0
votes
0
answers
225
views
Where can I get REDD dataset if http://redd.csail.mit.edu/ is not working?
Reference Energy Disaggregation Data Set (REDD) is used for research in Non-intrusive Load Monitoring (NILM). Every paper and blogs points towards http://redd.csail.mit.edu/ as data downling link. ...
- 103
1
vote
2
answers
99
views
How to decide the most suitable technique to handle Class Imbalance
For handling an imbalanced dataset, we have a variety of techniques like adjusting class weights, oversampling, undersampling, SMOTE and its different variations (RCSMOTE, GSMOTE, DBSMOTE).
My ...
- 173
1
vote
1
answer
57
views
If I'm comparing performance between two different datasets should sample and class size be uniform?
If I'm comparing performance between classification models on two different datasets should the number of samples per class, the number of classes, and features per sample be the relatively the same ...
- 13
0
votes
0
answers
23
views
How to reference an intermediary object in scala?
I am using spark/scala to do some transformations on a spark dataframe. I have the following code:
...
- 11