Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
485
questions with no upvoted or accepted answers
4
votes
1
answer
78
views
How to collect a computer vision dataset?
Context: my team is researching some plant-growth patterns using object detection models. We have collected a small set of images (a few thousand) and ran some simple experiments. The results were ...
- 141
4
votes
1
answer
4k
views
Weighted loss functions vs weighted sampling?
For image classification tasks, is there a practical difference between using weighted loss functions vs. using weighted sampling? (I would appreciate theoretical arguments, experience or published ...
4
votes
1
answer
160
views
Relation between an underlying function and the underlying probability distribition function of data
I heard and read a lot of times the following statements and got a lot of confusion over time.
Statement 1: The goal of machine learning is to get a function from
the given data
Statement 2: The goal ...
- 157
3
votes
0
answers
51
views
Public data set with exciting data
I'm going to give a talk about a tool for viewing tabular datasets. I'll first give this talk at a meetup, then hopefully at a series of conferences over the next few years.
I want to demonstrate to ...
- 255
3
votes
0
answers
57
views
Is it Scientifically Rigorous to create a multimodal ML Model with data from multiple sources
I'm attempting to create a multimodal machine learning model for disease diagnosis. However, I'm having quite a bit of difficulty finding public data sets with all the data I need.
For example, I need ...
3
votes
0
answers
35
views
Dataset availability for automatic text summarization
I'm working on an automatic text summarization NLP problem and looking for a dataset with USA legal case reports similar to the Australian legal case reports dataset in UCI repository.
Can you please ...
- 379
3
votes
0
answers
23
views
Why RANDOM noise images always predicted as BIRD?
Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%.
However when feeding into 5000 random noise images (Gaussian noise with the ...
- 31
3
votes
2
answers
2k
views
How to load numerous files from google drive into colab
I am trying to load in 30k images (600mb) from Google drive into Google Colaboratory to further process them with Keras/PyTorch.
Therefore I have first mounted my Google drive using:
...
- 89
3
votes
0
answers
309
views
How do I implement masking in TensorFlow eager execution?
I am training a stateful RNN on variable length sequences (optional: see my previous question for more details).
I padded the sequences to a fixed length with the value -1.
The when batches are ...
- 293
3
votes
0
answers
77
views
Can you have too uniform test data in a feedforward neural network?
I have been playing around trying to implement my own feedforward neural network. To try it out I decided on an easy example. 3 inputs, 3 output. When you send in ...
3
votes
2
answers
513
views
Genetic neural network to satisfy variable number of inputs and outputs
I have what I propose as a solution to my problem, however I haven't ever seen it mentioned in this way, so I worry that there is a valid reason not to do things this way.
I have a dataset of > 100,...
- 129
3
votes
0
answers
397
views
global flight network (with number of pessengers per year) dataset
Is there such a global flight network dataset that gives you:
flight route (the connection between each airports or cities by some flight company)
the number of passengers (per year) taking each ...
- 163
3
votes
0
answers
365
views
Where can I find a >1K dataset of annotated store receipt pictures/scans?
I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice:
vendor
date of purchase
total amount
(...
- 131
3
votes
1
answer
58
views
Am I supposed to be using Mixed Effects?
I ran a GLS random effects regression on some NBA data in Stata, and I was told that it was wrong because I didn't use mixed effects model. This may every well be the case, but I was quite confused by ...
- 31
3
votes
0
answers
299
views
Are SAS Data Storage Options designed for Big Data?
I understand that SAS has some data storage options in the form of
Default Storage tables or SAS datasets - in the SAS library.
Multi-dimensional Storage cubes - in the OLAP Server.
Third Party ...
- 795
2
votes
0
answers
47
views
How much data and computation power do I need to train a machine translation model using Transformer architecture?
I am working right now on creating a dataset to use in creating a machine translation model to translate between two dialects. I have two questions that I am trying to find an answer for:
How much ...
- 21
2
votes
0
answers
1k
views
How many samples in dataset are required to fine-tune BERT for binary classification?
I'm trying to fine-tune a BERT-based model for a binary classification task (data is in English). The dataset I'm working with is quite small (~500 samples, out of which 80% are currently used for ...
- 21
2
votes
1
answer
33
views
Return the gradient and y intercept (m, b) to create two lines to best fit the data
I have been working on this task for a few hours now and have been unsuccessful with getting the target result. I have tried using multiple methods of trying to split the dataset using different ...
- 21
2
votes
2
answers
555
views
How to get sentiment score for a word in a given dataset
I have a sentiment analysis dataset that is labeled in three categories: positive, negative, and neutral. I also have a list of words (mostly nouns), for which I want to calculate the sentiment value, ...
- 21
2
votes
0
answers
17
views
Addressing class imbalance: coefficient in loss vs threshold on confidence
Let's say I am training a neural network for a binary classification task, with classes A and B, which is trained using gradient descent with a cross-entropy loss function. The dataset is imbalanced, ...
- 21
2
votes
1
answer
108
views
Where can I find free multi-instance single-label datasets for object detection?
I'm trying to find free multi-instance single-label datasets for object detection online.
By "multi-instance and single-label" I mean that each image contains only object belonging to one ...
- 71
2
votes
0
answers
27
views
Best Method for Using Experts to Score Qualitative Data
TL;DR: Looking for recommendations or resources on the best method to have subject matter experts score qualitative data to train ML model.
Problem:
I am working on a problem in the biochemistry ...
- 21
2
votes
1
answer
263
views
Training a neural network with TWO possible correct outputs for one input
I have a system as a black box that has two correct outputs for a single input sample.
now I want to train a neural network to generate at least one of the correct outputs for that input sample.
what ...
2
votes
1
answer
36
views
Given a regression based model with many feature variables; what tools would you utilize to figure out which feature variables add the most variance?
Given a hypothetical dataset {S} with 100 X feature variables and 10 predicted Y variables.
X1
...
X100
Y1
....
Y10
1
..
2
3
..
4
4
..
3
2
..
1
Let's say I want to improve the accuracy of Y1. I am ...
2
votes
0
answers
49
views
Dataset for simultaneous object detection, 6D pose estimation and grasp detection
I am trying to find a dataset which meets any of the following requirements.
Dataset of RGB or RGBD images with object bbox annotations, 6d pose annotations and grasp annotations.
Dataset of RGB or ...
- 21
2
votes
1
answer
33
views
Architectures that take inputs of mixed sampling rates
Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time ...
- 21
2
votes
1
answer
104
views
What is the difference between Pachyderm and Git?
I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:
It holds all your data in a central accessible ...
- 516
2
votes
1
answer
5k
views
How to load all images using image_dataset_from_directory function?
I am working on a multi-label classification problem and faced some memory issues so I would to use the Keras image_dataset_from_directory method to load all the ...
- 381
2
votes
1
answer
27
views
How will my CNN results be affected by large discrepancies between the number of samples in some of the classes?
The number of samples in my dataset range between 3800 and 100,000 per class. Was wondering if my neural network will be more biased towards the classes with a higher number of images. I'm trying on ...
2
votes
1
answer
4k
views
How to resize MNIST images to fit AlexNet model
I am using the keras API to load in the MNIST dataset. My problem is I need to use AlexNet as my algorithm. Understanding the AlexNet model, I require to start with 277x277 images but the MINST ...
- 21
2
votes
0
answers
121
views
Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values
Context:
My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.).
It is too costly for us to emulate agents. We ...
- 21
2
votes
1
answer
168
views
Size of datasets over years
I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years.
I just found the following statistics:
The poll of KDnuggets that actually shows that over ...
- 133
2
votes
1
answer
834
views
Generating synthetic data based off existing real data (in Python)
I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future ...
- 780
2
votes
2
answers
229
views
Log analysis dataset with labeled cybersecurity issues
I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
- 21
2
votes
1
answer
154
views
How to preprocess an ordered categorical variable to feed a machine learning algorithm?
I have a categorical variable that measures the income of a family:
A: no income
B: Up to $500
C: $500-$700
…
P: $5000-$6000
Q: More than \\\$6000
It seems odd to ...
- 21
2
votes
2
answers
829
views
How can I use my own dataset for Image segmentation using Tensorflow
I have a huge problem using my own created dataset for image segmentation using Tensorflow. The dataset that I've build contain images like the one shown below:
The problem that I have is: How do I ...
- 21
2
votes
3
answers
2k
views
Which algorithm to use for transactional data
I'm given a Dataset of transactions and asked to find insights for businesses. I'm extremely new to ML / Data science and have only been experiencing with KMeans. The dataset has the following ...
- 21
2
votes
0
answers
311
views
Will setting up time series data in this way cause data leakage?
I am trying to predict future stock market values using a gradient boosted tree model. As far as I know, gradient boosted trees use the data in one row, and only that row, to predict the target ...
- 197
2
votes
0
answers
11
views
Dataset suggestion: video dataset with accurate depth and pose info
Please suggest some datasets (can be synthetic, like rendered in blender or so) which have video frames (30fps preferred) with accurate ground truth depth maps and pose (rotation and translation for ...
- 724
2
votes
0
answers
81
views
Regression dataset with categorical features
I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties:
Be a tabular dataset (no images).
Have at least 20k rows,...
- 6,031
2
votes
1
answer
194
views
What's an appropriate datastore for variable length sequence data for PyTorch consumption?
I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats.
I need a datastore that can rapidly ...
- 121
2
votes
1
answer
80
views
What to do if a specific label of a category appears only a few times?
Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
user83221
2
votes
0
answers
19
views
Combining Aggregated Dependent Data with Granular Independent Data
As data science newbie, I thought I was on the right tract until something doesn't seem right after plotting. A simplified structure of my dataset is as follows:
FIPS (Census Tract ID)
Drug-related ...
- 21
2
votes
1
answer
168
views
News de duplication dataset
I am looking for a news dataset with semantically duplicate news articles tagged.
Basically all the news articles which talk about the same story should be grouped. The stories can be worded ...
- 21
2
votes
2
answers
95
views
How to gather training data for simple voice commands?
I'm trying to build a machine learning model for recognizing simple voice commands like up, down, left, etc.
On similar problems based on images, I'd just take the picture and assign a label to it.
...
2
votes
0
answers
52
views
Fuzzily join two large sets of postal addresses
I have two tables of postal address information - the one is about 2 million records, the other roughly 40 million. They have quite bad quality, and also are not quite compatible with each other (...
- 121
2
votes
0
answers
74
views
Keras Binary Classification val_acc won't go past ~67; Full data and code included
I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
- 33
2
votes
2
answers
142
views
How can I detect patterns and/or keywords or phrases?
I am collecting data in a database via php from apache.
I am interested in detecting patterns in each column for now.
For example manual examination of the data shows the pattern phpmyadmin is ...
- 121
2
votes
0
answers
21
views
Where should we release our video dataset for activity recognition?
We have an activity recognition dataset made of tens of thousands video clips.
Which are the alternatives to release it to the research community?
Is there any public website?
I saw for instance ...
2
votes
1
answer
209
views
Dataset where svm performance is significantly different from random forest
Is there a specific dataset where svm performs significantly better or worse than random forest?
I know that the performance could depend on the dataset but is there a specific dataset?
