Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,496
questions
0
votes
0
answers
12
views
Should I log transform the data if the dataset is small?
Suppose I have a dataset with only about 25 rows.
All the relevant columns are all integers. The ranges of them go from 1 million to 100 million, and the distributions are all skewed right.
In this ...
- 1
0
votes
0
answers
8
views
How to make duplicates unique in a DataFrame robustly?
I have the following table.
I was able to parse it and display it as a DataFrame with pandas in python
I need to change the Ref for every value that is duplicate such that it becomes unique.
For ...
- 101
0
votes
1
answer
15
views
How should I plot 2D data as 3D plot?
I got data (1503, 38697) (cell(datapoint), genes(features)), each elements has integer values (gene expression).
so something like
...
0
votes
0
answers
24
views
How can I read hdf5 files stored as 1-D array. and view them as images?
I have a large image classification dataset stored in the format .hdf5. The dataset has the labels and the images stored in the ...
- 111
0
votes
0
answers
17
views
Looking for datasets on automobile parts information for machine learning
I'm embarking on a machine learning project that requires a comprehensive dataset of automobile parts information. The goal is to train a model that can identify and categorize various auto parts, ...
- 1
0
votes
0
answers
6
views
How to get / calculate statistics about voxceleb2?
VoxCeleb2 contains ~1M utterances from ~6000 speakers (multilingual).
I want to get statistics about voxceleb2:
number of utterances & speakers for each language
I tried to find the metadata ...
- 363
0
votes
0
answers
17
views
Seeking datasets for training a Language Model on U.S. mortgage loan processes
I'm in the process of training a Language Model (LLM) and require datasets that encompass various aspects of the U.S. mortgage loan process. The model's aim is to understand and simulate decision-...
- 1
0
votes
0
answers
26
views
propper feature encoding
I working with the following data set
also here is it's detailed description of "packet_dat" column
I can't understand how I can encode packet_dat column into proper feature so my ...
- 101
0
votes
0
answers
11
views
Packet data needed fot the model
I want to use this data set https://huggingface.co/datasets/rdpahalavan/network-packet-flow-header-payload for network attack classification
Here is packet structure from the README file.
Assuming I ...
- 101
0
votes
1
answer
47
views
Proper way to build time series dataset
I have daily time series but need to predict weekly total.
Here is example:
date
col0
col1
col2
col3
col4
target
0
2023-01-01 00:00:00
-1.021
0.708
-0.564
-0.502
0.244
-106.538
1
2023-01-02 00:00:...
- 1
0
votes
0
answers
14
views
Feature selection on datasets with both categorical and numerical features
I'm proposing a novel methodology for feature selection in the context of tabular datasets that contain both numerical and categorical features. In order to prove the efficacy of my methodology, I ...
0
votes
0
answers
22
views
statistical analysis of two data sets have different number of sampling
I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
- 1
0
votes
0
answers
17
views
How to make a dataset for Audio Speech Recognition of Arabic Language
I am trying to map Quranic Verses for the identification of errors. Different audios for the same verse have different durations as it depends on the reciter. I used Audacity to normalize but the ...
- 1
0
votes
0
answers
16
views
Open Source Model or dataset for attention map visualization
Helloo (:
I'm trying to develop a python application where users can upload an image e.g. of their website and receive a heatmap or attention map in return. This map should indicate where users are ...
- 1
0
votes
0
answers
14
views
What do I need to take into account if I want to test one model on different test datasets?
I'm just new to machine learning, I want to test a generative model on two different test datasets to compare its performance on different datasets. What do I need to take into account about the ...
- 1
0
votes
0
answers
22
views
Model for small sample time series
Im a total novice and i need to estimate some kind of relation-proof model(Granger test results, correl matrix are already provide some evidence) with following dataset:
20 observations (2001-2021), 4 ...
0
votes
0
answers
30
views
Optimal ML classification approach
Background: I have an app data (impressions, user activities) that I can use as features for a multiclass classifier (5 classes).
I just want to discuss about some things that our team is having a ...
0
votes
1
answer
24
views
how to build model using two input dataset in which there is no common column to merge or combine
I want to create model for truck company in which trucks delivers the car for customers.i have two data sets. one is customer details like how many cars they want from particular area or terminal and ...
- 1
0
votes
1
answer
44
views
Very basic but how to understand data statistically for machine learning?
I’m trying to solidify my statistics so I really know how to use them in my analysis/models. However my concept of statistical testing gets completely messed up by context. I’m unsure defining exactly ...
0
votes
1
answer
26
views
Time Series Forecasting for Multiple Store Sales with Simultaneous Timestamps
I have a sales dataset with each store having a unique identifier. The dataset contains daily sales data for each store over a period of around two-years. I'm looking to build a time series ...
0
votes
0
answers
7
views
Systematic way of selecting internet texts for a machine translation corpus / dataset?
I am currently working on a neural machine translation project and want to gather a corpus (or dataset) of internet texts that are written in standard and plain language. In theory, it certainly makes ...
- 101
0
votes
0
answers
12
views
Seeking Guidance on Creating Sample Data for AI Business Use Case
Subject: Seeking Guidance on Creating Sample Data for AI Business Use Case
Dear StackExhange Family
I am currently working on an AI project aimed at providing AI embaded solutions to HealthCare, ...
1
vote
0
answers
22
views
Determining the start and stop of vehicle movement by GPS
There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
0
votes
0
answers
17
views
For Finetuning Llama 2, what form of data is required?
I am working on a customer service chatbot project. I have files of different product's manuals on which i need to train the LLM which is Llama 2. As these are manuals so they not in Q/A form. In this ...
0
votes
0
answers
14
views
pdf dataset - personal information publicly available for unredaction project
I'm working on a research project, which is focused on demonstrate ineffectiveness of document redaction methods.
I'm looking for a dataset of pdf documents which contains generalities about real ...
- 1
0
votes
0
answers
28
views
divide a large group of people into subgroups based on two parameter
note in advance: I'm new to data-analysis and although my major civil engineering taught me about statistics, I did not apply it the way I would have encountered in real life, or in this field of work....
0
votes
1
answer
26
views
Binary Classification of Images- CNN
I am learning ML and am working on a CNN problem where I need to classify images of CATS and DOGS.
The way I have setup the labels is that cats are 1 and dogs are 0. I have made the final output layer ...
0
votes
0
answers
15
views
RMSE of whole and part of test dataset
Can anybody help me to understand the behavior of metrics (RMSE namely) when testing model? I have NN with 1 hidden layer for regression task. RMSE equal 0.07 for external test dataset. But if I break ...
- 1
0
votes
0
answers
7
views
for multi-tasking training should I use big datasets as one file or each table as one csv?
Assuming we want to create a dataset of books.
There would be dataset A: id_book, title, Author, Subject, Topic, Body [paragraph 1,2,3], Sites ...
I was wondering what if i split each book into one ...
0
votes
0
answers
25
views
No answers on gpt.h2o.ai if change to "personal MyData Collection" on the platform
I have downloaded pre-trained models BERT, llama-2 and some others (for CPU). I wanted to test, on the h2ogpt platform what output would each provide.
My problem is how the knowledge of the models ...
0
votes
0
answers
7
views
how to convert these string columns so that i can put the data to the regressor
https://drive.google.com/file/d/1yQR1xJvzWd2UGgGQS2bHhdywZEE_Nu0J/view?usp=sharing
I have to change the name of the columns: city, batting_team, bowling_team, batsman, non_striker, bowler. These ...
1
vote
1
answer
36
views
Alternative Source for Downloading OTB Dataset for Visual Object Tracking
I'm interested in downloading the OTB dataset for visual object tracking. The official site (http://cvlab.hanyang.ac.kr/tracker_benchmark/index.html) appears to be down. Does anyone know of ...
- 13
1
vote
1
answer
27
views
A good set of datasets/models for testing an NLP technique
I am a machine learning researcher who up until this point has primarily worked on Computer Vision problems. However, I have an idea for an NLP technique involving a novel Transformer architecture, ...
- 13
0
votes
0
answers
19
views
How can I perform an analysis of the NHIS imputed income variables?
I have downloaded five family income variables from https://nhis.ipums.org/nhis-action/variables/group?id=economic_income (INCPPOINT1, INCPPOINT2, INCPPOINT3, INCPPOINT4, INCPPOINT5) for they years ...
1
vote
2
answers
91
views
Cluster/Similarity problem with two datasets of different cardinality
I want to cluster financial products according to their similarity. I have two dataset of different cardinality:
One-to-One dataset: One ID has One attribute/feature per column - Describes a ...
- 550
0
votes
0
answers
17
views
Expanding torch dataLoader by repeating elements (or something equivlent)
I have a pytorch classifier with stochastic elements. For each MNIST minibatch, I would like to feed it in K times (from the same starting state), and then average the resulting weight updates in ...
- 101
0
votes
0
answers
7
views
Terminology: What is it called when the filter criteria is added to the data table?
When we want to focus on a group within our dataset, it is common to filter it.
Suppose we have a data table showing how many points 4 players earned in a game.
Table 1: Game Scores for 4 Players
Age
...
- 123
0
votes
0
answers
9
views
Create Dataset (ADS) to build predictve model for servey data
I have collected labeled survey data spanning 8 months in 2023, capturing satisfaction levels ranging from "very satisfied" to "very dissatisfied."
My objective is to develop a ...
0
votes
0
answers
12
views
Datasets for demonstrating data/model/concept drift
Are there any well-known datasets that are useful for illustrating the concept of data and/or model/concept drift? I can create my own out of, say, MNIST (by e.g. removing digits and adding them in ...
0
votes
1
answer
36
views
How to use API Documentation of dataset
Some websites provide a link to a dataset (in Excel sheets format) which allows the dataset to be downloaded. But some others additionally provide API documentation, like this site. Can you please ...
- 103
0
votes
1
answer
18
views
Object Classification Dataset Creation
There is a problem I've faced recently which I'm not sure my approach is proper or not. There is bunch of field videos which I run a semi-supervised detection model to extract crops to train my ...
- 38
0
votes
2
answers
34
views
Optimal Data Split
I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, ...
0
votes
2
answers
33
views
Lack of imagination when creating projects
Whenever I find a set of data to carry out a project, I end up running out of ideas and don't know how to proceed with the project. I don't know how to proceed or what information to look for. How do ...
0
votes
1
answer
40
views
different range of target values in neural network
I am working on a neural network regression code. The dataset includes 14 features in the range value between -1 and 1. while the target variable is changing among (0.000759) to (1100).
The target ...
- 1
0
votes
0
answers
10
views
How should I save data in deeplearning with nosql (mongodb)?
I usually use file system to manage data for my deep learning model, but one of my boss told me to make nosql database to manage data.
Datasets I use have m rows, and n columns of count matrix and ...
1
vote
1
answer
50
views
How to know the confidence of a classification on unlabeled data generated after model training?
I have created (in python) the code for a Random Forest classification model for a labeled dataset using sklearn. The model works very well.
...
0
votes
1
answer
32
views
Best strategy for handling missing groups of features
I am currently working on a ML problem where the features used for modelling are sourced from different places/providers. It is very unlikely to find the features from all the different sources to be ...
0
votes
0
answers
18
views
What distribution-shift datasets exist for ML?
What datasets there are which contain a (labelled) distribution shift or which distribution-shifted datasets share classes?
Currently I have found some interesting results using CIFAR10 and the ...
- 1
0
votes
0
answers
27
views
Open source dataset (manufacturing, machine operations)
I am looking for an open source dataset from the manufacturing domain (sensor data, time series) with specific traits. It should stem from a process consisting of a sequence of distinct machine ...
0
votes
0
answers
24
views
Acquiring Ground Truth for ISIC 2019 Challenge Test Set for Publication Purposes
I have been working with the dataset from the ISIC 2019 challenge, which is not live anymore. I was hoping to find the ground truth data for the test set published on the official website for further ...
- 249