Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,477
questions
0
votes
0
answers
9
views
A good set of datasets/models for testing an NLP technique
I am a machine learning researcher who up until this point has primarily worked on Computer Vision problems. However, I have an idea for an NLP technique involving a novel Transformer architecture, ...
0
votes
0
answers
15
views
How can I perform an analysis of the NHIS imputed income variables?
I have downloaded five family income variables from https://nhis.ipums.org/nhis-action/variables/group?id=economic_income (INCPPOINT1, INCPPOINT2, INCPPOINT3, INCPPOINT4, INCPPOINT5) for they years ...
0
votes
0
answers
17
views
+50
Cluster/Similarity problem with two datasets of different cardinality
I want to cluster financial products according to their similarity. I have two dataset of different cardinality:
One-to-One dataset: One ID has One attribute/feature per column - Describes a ...
- 528
0
votes
0
answers
15
views
Expanding torch dataLoader by repeating elements (or something equivlent)
I have a pytorch classifier with stochastic elements. For each MNIST minibatch, I would like to feed it in K times (from the same starting state), and then average the resulting weight updates in ...
- 101
0
votes
0
answers
7
views
Terminology: What is it called when the filter criteria is added to the data table?
When we want to focus on a group within our dataset, it is common to filter it.
Suppose we have a data table showing how many points 4 players earned in a game.
Table 1: Game Scores for 4 Players
Age
...
- 123
0
votes
0
answers
9
views
Create Dataset (ADS) to build predictve model for servey data
I have collected labeled survey data spanning 8 months in 2023, capturing satisfaction levels ranging from "very satisfied" to "very dissatisfied."
My objective is to develop a ...
0
votes
0
answers
8
views
Datasets for demonstrating data/model/concept drift
Are there any well-known datasets that are useful for illustrating the concept of data and/or model/concept drift? I can create my own out of, say, MNIST (by e.g. removing digits and adding them in ...
0
votes
1
answer
33
views
How to use API Documentation of dataset
Some websites provide a link to a dataset (in Excel sheets format) which allows the dataset to be downloaded. But some others additionally provide API documentation, like this site. Can you please ...
- 103
0
votes
1
answer
17
views
Object Classification Dataset Creation
There is a problem I've faced recently which I'm not sure my approach is proper or not. There is bunch of field videos which I run a semi-supervised detection model to extract crops to train my ...
- 38
0
votes
2
answers
26
views
Optimal Data Split
I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, ...
0
votes
2
answers
29
views
Lack of imagination when creating projects
Whenever I find a set of data to carry out a project, I end up running out of ideas and don't know how to proceed with the project. I don't know how to proceed or what information to look for. How do ...
0
votes
1
answer
22
views
different range of target values in neural network
I am working on a neural network regression code. The dataset includes 14 features in the range value between -1 and 1. while the target variable is changing among (0.000759) to (1100).
The target ...
- 1
0
votes
0
answers
9
views
How should I save data in deeplearning with nosql (mongodb)?
I usually use file system to manage data for my deep learning model, but one of my boss told me to make nosql database to manage data.
Datasets I use have m rows, and n columns of count matrix and ...
1
vote
1
answer
31
views
How to know the confidence of a classification on unlabeled data generated after model training?
I have created (in python) the code for a Random Forest classification model for a labeled dataset using sklearn. The model works very well.
...
0
votes
1
answer
30
views
Best strategy for handling missing groups of features
I am currently working on a ML problem where the features used for modelling are sourced from different places/providers. It is very unlikely to find the features from all the different sources to be ...
0
votes
0
answers
13
views
What distribution-shift datasets exist for ML?
What datasets there are which contain a (labelled) distribution shift or which distribution-shifted datasets share classes?
Currently I have found some interesting results using CIFAR10 and the ...
- 1
0
votes
0
answers
18
views
Open source dataset (manufacturing, machine operations)
I am looking for an open source dataset from the manufacturing domain (sensor data, time series) with specific traits. It should stem from a process consisting of a sequence of distinct machine ...
0
votes
0
answers
10
views
Acquiring Ground Truth for ISIC 2019 Challenge Test Set for Publication Purposes
I have been working with the dataset from the ISIC 2019 challenge, which is not live anymore. I was hoping to find the ground truth data for the test set published on the official website for further ...
- 249
0
votes
1
answer
46
views
How would normalizing be affected by outliers? And how to avoid it?
I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction.
I'm trying to find the best way to rank all suppliers ...
- 1
0
votes
1
answer
30
views
Better understand the scatter plots
I'm trying to understand how to read a scatter plot based on the results of my confusion matrix. This is the CM:
I saw an high correlation between column 5 and column 6, a low correlation with column ...
- 59
0
votes
1
answer
24
views
Request for Assistance with Acquiring a Diverse Cry Dataset for Research Project
I am working on a research project aimed at classifying babies' cries based on their needs. However, I have encountered difficulties in obtaining a suitable cry dataset.
The only dataset I was able to ...
0
votes
3
answers
29
views
Data preprocessing
I just want to know how to determine whether to remove the missing values or to impute them with mean , median or mode. I usually remove the missing values but it decreases the size of the dataset by ...
0
votes
1
answer
30
views
Multilabel classification: Choosing threshold
I'm creating a multilabel classification approach based on sentence embeddings applied to text taken from a chatbot.
We have the following:
a training dataset of 2,500 lines, where each line is a ...
1
vote
0
answers
64
views
Annotating and Structuring a Dataset for Duplicate Detection
I'm currently working on a project that requires the detection of duplicate bands in Western blot images. The task involves two types of duplicates:
...
- 111
0
votes
0
answers
11
views
How to subset and approximate a private dataset's characteristics?
Assume an Object Detection task. The private dataset has its bounding-box (bbox) characteristics, and assume only width-height measures are relevant.
Assume we know what is that width-height ...
- 1
0
votes
1
answer
12
views
Classification Problematics : Feature Number Variance & Feature Repetition
I have a harsh case study (in my mind).
The problem is I need make binary classification on Quality of Service (good or bad). I have a feedback on quality on groups of devices belonging to company. I ...
- 1
0
votes
0
answers
7
views
I want to import my states legislation into a database. I need advice in parsing large HTML, TXT into an importable form. It is Large PDF initialy
My data consists of large pdf files containing state legislation. I can convert the pdf to xml, txt or html, and I can manually go through, identify the tables and write a script to extract only the ...
- 1
0
votes
0
answers
66
views
Best practice for fine tuning LLM
I have a dataset that i have collect for specific topic
the dataset is in these format :
raw text (similar to shake spare dataset) where it has no label or input, just text
Question and answer ...
- 185
0
votes
1
answer
64
views
Can bert uncased predict text classification on foreign data?
I am trying to do the fake news/real news classification and used a pre-trained bert uncased model as transfer learning and it gave a solid 81% accuracy. But the problem is while doing sanity checks, ...
0
votes
0
answers
6
views
Dataframe shape for multi-point prediction using LSTM & time-serise data
Can I ask a very simple question about LSTM time-series data prediction?
I have 1258 data points which are Amazon's historical stock price by day unit.
And I build the below data structure in order to ...
- 113
0
votes
0
answers
40
views
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python - from_gerator fails
I know this question has answers, but they are not working.
why this fails ?
(and why tensorflow has troubles in reading numpy!!)
...
- 165
0
votes
0
answers
25
views
How do I test with custom images for a model that was trained on quick draw npy dataset
I have been trying to test my CNN model on a custom doodle of an apple that I drew. But even when I preprocess the image to have the same shape with the training data, the model gives wrong prediction ...
- 1
2
votes
1
answer
38
views
Smart Selection of Training Data for Fine-tuning Language Models in Small Domains
Background
I am working to make language models (for example, Stanford's Alpaca model) perform well on a new small domain through fine-tuning on domain-specific dataset $D$.
If the size of $D$ is $N$, ...
- 173
0
votes
0
answers
12
views
Do Hospital Episode Statistics (HES) data (NHS UK) contain labs?
Can anyone tell me if HES data contain lab results such as blood tests commissioned and their result? That can be in any one of the HES datasets.
Thank you.
- 101
0
votes
1
answer
26
views
Spam tweets dataset needed [closed]
Where can I find labelled spam tweets dataset that has at least 50000 rows. I've searched everywhere; but there are very few datasets available, and those datasets are way too small.
I am in urgent ...
0
votes
1
answer
62
views
Open retail/ecommerce dataset with customers, transactions, products, loyalty, discounts, promotion information?
Does anyone know of an open retail/e-commerce dataset with information about customers, transactions, products, loyalty, discounts, and promotion information?
I am planning to create 360-degree ...
- 101
2
votes
2
answers
150
views
Training GPT Model with Swagger Documents, Need Help with Model Fit
We are currently developing an application that performs actions based on user input using 'gpt-3.5-turbo-0613'.
For example, if a user enters the prompt "create a user with the name Potter and ...
- 23
1
vote
2
answers
37
views
Tips for scraping crypto data in the right way
I am scraping data from crypto site and want to use neural network algorithm for predicting data.
the way i save data is like these:
and there is bunch of other features like open/high/low/close for ...
0
votes
0
answers
15
views
Multiple coords to one data_vars in xr.dataset
I am trying to create a xarray dataset for a list of events (labeled 1 or 0, so it's an array of integers). To each event three numbers need to be attached - a date, longitude, and latitude.
I provide ...
- 125
1
vote
0
answers
24
views
Is there a statistical test for comparing my dataset percentages with published percentages?
I have a clinical dataset which is showing the percentage of A is 57% and B is 35% for where an tumours is initially found. However the literature is showing it is A is 73% and B is 23%.
Is there a ...
- 11
-1
votes
2
answers
92
views
How to generate synthetic dataset with dependent and independent features?
I want to generate a synthetic dataset, in python, composed of 10 features. 5 are dependent among each others and 5 are independent.
Generate dependent features.
...
- 57
0
votes
2
answers
68
views
Is there any rationale for performing SMOTE-ENN before train-test-split?
I have created a classification model for predicting data, and the problem is that the two classes are highly imbalanced
I have a problem. I have created a classification model for predicting data, ...
0
votes
1
answer
28
views
How to handle ID variables when splitting data for machine learning?
I'm new to machine learning, and I'm working with some international head-to-head sports competition data. I used relational data creation techniques in tidyverse to join several data sources to ...
- 1
1
vote
0
answers
36
views
Accounting for differences in average teacher salaries, adjusted for inflation and the cost of living in each state
My aim is to determine whether teacher salaries in all 50 of the United States over between 2013 and 2023—adjusted for inflation and the cost of living—differ significantly. I would like to ask what ...
- 11
0
votes
0
answers
19
views
Problem with the size of test data?
I am working on my thesis, which has two different research questions:
Evaluate transformer models while incorporating non textuel features
Evaluate the importance of data quality in tranformer ...
- 1
0
votes
0
answers
12
views
I'm looking for a dataset on enviroment monitoring
I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
- 1
0
votes
2
answers
74
views
How to create a plot of specific row with every column using python and which package to choose matplotlib or seaborn?
I'm trying to create a plot for the ranking of each country from 2002 to 2023. I created this dataset by loading each csv file from the respective years, which contains the Countries and Ranking ...
0
votes
1
answer
43
views
Data transformation pipeline error
When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
- 1
0
votes
1
answer
22
views
Domain Constraints
Consider a dataset with 5 numerical features: A, B, C, D and E. Can we train a deep learning/machine learning model which can learn constraints between (A,B) and (C,D,E) i.e I want to learn a function ...
0
votes
1
answer
56
views
Keras MLP not working
For my degree's final project I am working with Keras and trying to build different AI models. I'm having trouble with an MLP. First I preprocess the UNSW-NB15 dataset and later use it as input in a ...
