Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,491
questions
0
votes
1
answer
12
views
Classification Problematics : Feature Number Variance & Feature Repetition
I have a harsh case study (in my mind).
The problem is I need make binary classification on Quality of Service (good or bad). I have a feedback on quality on groups of devices belonging to company. I ...
- 1
0
votes
0
answers
8
views
I want to import my states legislation into a database. I need advice in parsing large HTML, TXT into an importable form. It is Large PDF initialy
My data consists of large pdf files containing state legislation. I can convert the pdf to xml, txt or html, and I can manually go through, identify the tables and write a script to extract only the ...
- 1
0
votes
0
answers
78
views
Best practice for fine tuning LLM
I have a dataset that i have collect for specific topic
the dataset is in these format :
raw text (similar to shake spare dataset) where it has no label or input, just text
Question and answer ...
- 185
0
votes
1
answer
160
views
Can bert uncased predict text classification on foreign data?
I am trying to do the fake news/real news classification and used a pre-trained bert uncased model as transfer learning and it gave a solid 81% accuracy. But the problem is while doing sanity checks, ...
0
votes
0
answers
7
views
Dataframe shape for multi-point prediction using LSTM & time-serise data
Can I ask a very simple question about LSTM time-series data prediction?
I have 1258 data points which are Amazon's historical stock price by day unit.
And I build the below data structure in order to ...
- 113
0
votes
0
answers
113
views
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python - from_gerator fails
I know this question has answers, but they are not working.
why this fails ?
(and why tensorflow has troubles in reading numpy!!)
...
- 165
0
votes
0
answers
27
views
How do I test with custom images for a model that was trained on quick draw npy dataset
I have been trying to test my CNN model on a custom doodle of an apple that I drew. But even when I preprocess the image to have the same shape with the training data, the model gives wrong prediction ...
- 1
2
votes
1
answer
42
views
Smart Selection of Training Data for Fine-tuning Language Models in Small Domains
Background
I am working to make language models (for example, Stanford's Alpaca model) perform well on a new small domain through fine-tuning on domain-specific dataset $D$.
If the size of $D$ is $N$, ...
- 83
0
votes
0
answers
17
views
Do Hospital Episode Statistics (HES) data (NHS UK) contain labs?
Can anyone tell me if HES data contain lab results such as blood tests commissioned and their result? That can be in any one of the HES datasets.
Thank you.
- 101
0
votes
1
answer
39
views
Spam tweets dataset needed [closed]
Where can I find labelled spam tweets dataset that has at least 50000 rows. I've searched everywhere; but there are very few datasets available, and those datasets are way too small.
I am in urgent ...
0
votes
1
answer
189
views
Open retail/ecommerce dataset with customers, transactions, products, loyalty, discounts, promotion information?
Does anyone know of an open retail/e-commerce dataset with information about customers, transactions, products, loyalty, discounts, and promotion information?
I am planning to create 360-degree ...
- 101
2
votes
2
answers
256
views
Training GPT Model with Swagger Documents, Need Help with Model Fit
We are currently developing an application that performs actions based on user input using 'gpt-3.5-turbo-0613'.
For example, if a user enters the prompt "create a user with the name Potter and ...
- 23
1
vote
2
answers
41
views
Tips for scraping crypto data in the right way
I am scraping data from crypto site and want to use neural network algorithm for predicting data.
the way i save data is like these:
and there is bunch of other features like open/high/low/close for ...
0
votes
0
answers
15
views
Multiple coords to one data_vars in xr.dataset
I am trying to create a xarray dataset for a list of events (labeled 1 or 0, so it's an array of integers). To each event three numbers need to be attached - a date, longitude, and latitude.
I provide ...
- 125
1
vote
0
answers
25
views
Is there a statistical test for comparing my dataset percentages with published percentages?
I have a clinical dataset which is showing the percentage of A is 57% and B is 35% for where an tumours is initially found. However the literature is showing it is A is 73% and B is 23%.
Is there a ...
- 11
-1
votes
2
answers
135
views
How to generate synthetic dataset with dependent and independent features?
I want to generate a synthetic dataset, in python, composed of 10 features. 5 are dependent among each others and 5 are independent.
Generate dependent features.
...
- 57
0
votes
2
answers
99
views
Is there any rationale for performing SMOTE-ENN before train-test-split?
I have created a classification model for predicting data, and the problem is that the two classes are highly imbalanced
I have a problem. I have created a classification model for predicting data, ...
0
votes
1
answer
58
views
How to handle ID variables when splitting data for machine learning?
I'm new to machine learning, and I'm working with some international head-to-head sports competition data. I used relational data creation techniques in tidyverse to join several data sources to ...
- 1
1
vote
0
answers
38
views
Accounting for differences in average teacher salaries, adjusted for inflation and the cost of living in each state
My aim is to determine whether teacher salaries in all 50 of the United States over between 2013 and 2023—adjusted for inflation and the cost of living—differ significantly. I would like to ask what ...
- 11
0
votes
0
answers
19
views
Problem with the size of test data?
I am working on my thesis, which has two different research questions:
Evaluate transformer models while incorporating non textuel features
Evaluate the importance of data quality in tranformer ...
- 1
0
votes
0
answers
13
views
I'm looking for a dataset on enviroment monitoring
I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
- 1
0
votes
2
answers
180
views
How to create a plot of specific row with every column using python and which package to choose matplotlib or seaborn?
I'm trying to create a plot for the ranking of each country from 2002 to 2023. I created this dataset by loading each csv file from the respective years, which contains the Countries and Ranking ...
0
votes
1
answer
60
views
Data transformation pipeline error
When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
- 1
0
votes
1
answer
22
views
Domain Constraints
Consider a dataset with 5 numerical features: A, B, C, D and E. Can we train a deep learning/machine learning model which can learn constraints between (A,B) and (C,D,E) i.e I want to learn a function ...
0
votes
1
answer
62
views
Keras MLP not working
For my degree's final project I am working with Keras and trying to build different AI models. I'm having trouble with an MLP. First I preprocess the UNSW-NB15 dataset and later use it as input in a ...
0
votes
0
answers
13
views
Scaling datasets for multi-dataset time series
Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
- 137
0
votes
0
answers
19
views
Training a Face Recognition model using benchmark datasets
Is it recommend (or conventional) to train, for instance a face recognition method, using benchmark datasets such as XQLFW, RFW, CALFW, etc.? I would like to fine tune my model so it is more robust to ...
- 1
0
votes
0
answers
12
views
How to analyze 2 NLP datasets and understand their differences
I have 2 test sets (let's call them test set #1 and test set #2), and 2 Language models (let's call them model A and model B).
The issue I'm facing is that model A is performing better than model B on ...
- 1,004
0
votes
1
answer
130
views
What's the best approach to dealing with missing data in a dataset?
I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
- 1
0
votes
1
answer
30
views
how to prepare pandas dataframe from Solar Flare Data Set?
I am looking to download and use Solar Flare Data Set (https://archive.ics.uci.edu/ml/datasets/Solar+Flare) but unable to figure out how to get the pandas dataframe from the file that is being ...
- 101
0
votes
0
answers
20
views
How to manage and store datasets used to train Computer Vision models?
I am not sure if this is the right place to ask this question but I am looking for an insightful answer.
I have been working as a Data Scientist for one year now. Currently at the company that I work ...
0
votes
0
answers
27
views
Grouping data or clustering data
I have frame data of 10 subjects gathered from a timeseries analysis. Each subject have 100 frames except the reference frame (timeseries length is 100+1 = 101). Particular frame property is ...
0
votes
1
answer
48
views
Low-dimensional binary classification datasets
If one would like to study aspects of neural networks (say, in an academic paper), and would like to experiment on binary classification of vectors in low-dimensional space (say dim=2 or dim<6); ...
- 1
0
votes
0
answers
44
views
Multiprocessing data loading in colab
I want to convert mozilla common voice dataset from mp3 to wav. But this dataset is large and convertion takes many time. How can I make this convertion in colab with multiprocessing to decrease time ...
0
votes
1
answer
24
views
A guide to learn data analysis
I'm new to data analysis and I need to do a data analysis project using clustering methods for a course in R. I have no idea how to start and choose my data set. I'm looking for some resources. Is ...
- 101
0
votes
0
answers
16
views
Best way to regrid data (polar stereographic to regular)
I have data that's in 'polar stereographic' form and I want to regrid it to a regular grid that matches a grid I currently have. I've seen a few example that tend to be fairly complicated. I'd ideally ...
- 91
1
vote
1
answer
41
views
Can we create tensorflow or tflite model for object detection without uploading our dataset images to cloud?
I need to create a custom Tensor flow lite model for object detection to integrate in an android app. But I have a constraint that the dataset images to be used is confidential and cant be uploaded ...
0
votes
0
answers
22
views
Should I specify 'reference=train_set' when creating a evaluating lightgbm.Dataset?
According to the docs, we should offer a argument reference=train_set when creating a validating lightgbm.Dataset.
I know a little about the reason, i.e. there are ...
- 101
0
votes
1
answer
121
views
Does the Transformer model has memory to store the state accross different data injection sequences(segments)?
I've trained a transformer model based on the pytorch tutorial: https://pytorch.org/tutorials/beginner/transformer_tutorial.html,
But I found I've difficulties to understant this model's input and ...
- 113
0
votes
0
answers
9
views
is it sound practice to normalise & sum numerical variables to create a single metric for a prediction model? e.g's provided
Does this question aim to understand if this is sound practice? if not, I would appreciate a suggestion.
The goal is to use this metric:
regression
classification
as a metric
profile_views
...
0
votes
1
answer
172
views
Plot multiple catplots from a dataframe
I have a dataframe and I want to plot multiple catplots. I want to use a loop but i do not know how to code one for this. Any examples?
...
- 1
0
votes
0
answers
16
views
Having trouble manipulating data to make better dashboard
I have a test dataset which has has 39 columns. Index is given below:
Sample data is as follows:
Some columns are omitted due to privacy concerns. The colored cells in the index belong to one type. ...
1
vote
1
answer
44
views
I want to forecast a NetCDF data
I have a chlorophyl-a and sea surface temp (SST) in a NetCDF format, the data is 4 dimensional mattrix, each data is saved inside certain longitude and lattitude value, if the data plotted into ...
- 11
0
votes
0
answers
33
views
How should data from closed down sites or companies be licensed?
There are sites that are closed down but exists on archive.org, e.g.
Oyshee aka Rakuten Recipe (English) closed down, but the parent company Rakuten still exists
DP Review will close down in April ...
- 2,340
1
vote
1
answer
27
views
Comparing images in N channels
I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
- 125
0
votes
0
answers
19
views
Time series : add constants for each time series
I have N time series of shape (30*36) (time step, feature). For each time series 500 parameters that can be seen as the history of the time series have to be added. ...
- 29
0
votes
0
answers
25
views
How to label a transactions dataset made from scratch?
I have a question regarding creating a transactions dataset from scratch.
I've created customer profiles and am generating transactions based on these profiles. The way I do this is based on the ...
- 1
0
votes
1
answer
20
views
What is the best way to store large data when training a model?
This is rather a technical question, when training the model I import data locally to Google Colab runtime which does not seem to be efficient, when the runtime ...
- 101
0
votes
0
answers
15
views
How to estimate this variable in an MILP formulation
This is my first question being asked here. I've thought about different methods to do it, but to no avail. I want to estimate a variable that is either 0 or a positive number. Then I want to use this ...
0
votes
0
answers
9
views
Implementing a dataset to Computer Vision Article
I want to implement the PIE dataset in the AgentFormer arch.
AgentFormer uses ETH and nuScene datasets. I successfully run these datasets on this arch. However, I couldn't take a good way with the PIE ...
- 1