Skip to main content

Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Efficiently Implement a Data Transformation || Pytorch || coding advice

I have created the following class and function to create a 'custom' dataset. The use case was to sample every nth image from a data set to reduce the size of the data set while still maintaining ...
James's user avatar
  • 11
0 votes
0 answers
10 views

training data of stable cascade vs stable diffusion

So according to this: https://waxy.org/2022/08/exploring-12-million-of-the-images-used-to-train-stable-diffusions-image-generator/ Stable Diffusion was trained on data from CommonCrawler. I believe ...
x89's user avatar
  • 191
1 vote
0 answers
13 views

Recreating results from Research Paper

so I have been trying to recreate the results from this particular paper (Neural Collaborative Filtering). The dataset I use closely resembles this . I understand that I should my data into train and ...
Panos_42's user avatar
0 votes
0 answers
34 views

improve my f1_score for classification - pandas/sklearn

I would like advice on how to improve my f1_score for classification. I currently have something around 0.57. Dataset: lotWaferDie - lot, board and chip on which defects were measured string values ...
user162343's user avatar
0 votes
1 answer
13 views

Struggling with normalization/Standardisation for machine learning dataset

Sorry for what is probably a very obvious/rookie question. I'm currently doing a data science module for my degree and making very slow progress with the work. The case study i'm doing is around HR ...
Alex Ferry's user avatar
0 votes
0 answers
42 views

When is sampling bias acceptable?

Overview: Dataset is small and a bit messy and the task is to classify 5 classes wherein the targets are ordinal. Feature Engineering and Selection, Model Tuning, etc. did not produce acceptable ...
easymoneysniper's user avatar
0 votes
0 answers
7 views

What is the reference time point relative to which the capital-gain and capital-loss features of the UC Irvine Adults dataset are measured?

The Adults dataset available in the UC Irvine Machine Learning Repository is based on the 1994 census data (USA census, I presume). The dataset has two features named ...
Evan Aad's user avatar
  • 175
1 vote
0 answers
12 views

Are there any public medical image databases?

I'm not sure if I'm asking this in the right place, please direct me to the right place for me to ask this if I am not. I'm working on a project to look at images of the inner ear from an otoscope and ...
Skymero's user avatar
  • 11
1 vote
0 answers
7 views

Problem with HDI dataset

I am new to Orange3. I am trying to follow the same steps in the following article https://orangedatamining.com/blog/data-mining-covid-19-epidemics-part-1/ But unfortunately the HDI dataset that is ...
Sara Raad's user avatar
0 votes
0 answers
12 views

Building a dataset for classification

I'm thinking of building a powershell script classifier using different architectures of neural networks. I have approximately 6k powershell scripts (3k malicious, 3k benign). My questions are: How ...
freaks's user avatar
  • 1
1 vote
1 answer
19 views

Public Email Classification Dataset but not Spam vs Ham

Context Working to deliver a POC on automated email classification (in customer service context) to tag emails as related to feedback, complain, lost and found etc. The tags are not entirely exclusive,...
Della's user avatar
  • 335
0 votes
0 answers
8 views

Missing Value in a dataset

i'm curently cleaning a dataset and theres a column called " Institution", it is needed for encoding and training the classification model later so it needs to be cleaned. In that column, ...
Harry lou's user avatar
1 vote
1 answer
18 views

How to Balance Dataset extracted using image_dataset_from_directory

I'm new to tensorflow, so I've been trying to find the best way to do class balancing over a dataset where I used image_dataset_from_directory to load. But I haven't find the way to do it. I saw from ...
lopez-mgu's user avatar
0 votes
0 answers
14 views

How to select the subset which can represent the whole dataset

I have a time series of data for the whole year, which I need to run the analysis on Python. However, it takes a long time to run the model. I want to select a subset of data that can represent the ...
Thành Trần's user avatar
0 votes
0 answers
12 views

Machine Learning Algorithm for identifying the factors contributing to academic performance of students

I have a dataset with several qualitative and quantitative attributes, including age, location (longitude, latitude), city, parent occupation, family size, GPA etc. My task is to find the attributes/...
Dawood Ahmad's user avatar
2 votes
0 answers
23 views

Why is it called 'Test' dataset and not 'Testing' Dataset?

Training data is named as such as it is used for 'training'. Then why not the same for 'Testing' dataset? I apologise if this is not very relevant, but its something I believe only the community can ...
Sameervdtc's user avatar
0 votes
0 answers
9 views

SAS Macro: dereferencing a macro variable in the name of a column name of a dataset

In SAS dereferencing this does not work: ...
babipsylon's user avatar
0 votes
0 answers
7 views

Rename a variable (column name) in a SAS dataset with unknow suffix

The ICT team provides me with a dataset with a column that changes name every month, but always starts with "Reality". It's always column 10. There are other variables that match these ...
babipsylon's user avatar
0 votes
0 answers
8 views

Replacing values in a dataset based on a IF statement

I have code for which this works perfectly, see code 1 below. But for code 2, it does not work at all. Very strange. Only difference is the way the tables were created, and the use of macro variables ...
babipsylon's user avatar
0 votes
0 answers
15 views

SAS: how to replace a value in column two in a dataset for which column one = x?

So, total SAS noob here (and I don't like it al all so far). I tried 4 different methods, which all do not work. So I guess in all of these code, something similar is wrong, something rather ...
babipsylon's user avatar
0 votes
0 answers
12 views

Dataset for access control policies?

I'm working on a dynamic access control policy generation model using machine learning and I need a real dataset to evaluate the performance. Is there any access control policy dataset consisting ...
Binod Karunanayake's user avatar
0 votes
0 answers
9 views

Why are Delta 15 N and Delta 13 C not included in the simplified Palmer Penguin dataset?

I was looking at the Palmer Penguin dataset and noticed that it had two versions, one simplified and the other raw. While I understand why columns such as Comments ...
Hippopotoman's user avatar
0 votes
0 answers
32 views

Best practises for creating datasets for the purpose of finetuning LLMs

I am working on a problem for which no datasets exist. I have obtained several examples from this domain, and so far have been using them in Large Language Model (LLM) prompts(few shot learning) but I ...
Karl 17302's user avatar
0 votes
0 answers
9 views

Training split generation - Extremely Slow

I am fine-tuning mbert on wikipedia dataset, loaded with Datasets (Hugging face) ...
GaiusGhislerianus's user avatar
0 votes
0 answers
16 views

Does hours spent training early in an Associates career impact the amount of revenue they generate in the future?

I can use some help with project design and choosing the correct method to answer this question. I have a dataset that contains associate ID, associate specialization, hours worked, revenue, period, ...
Danielle DeLosa's user avatar
0 votes
1 answer
11 views

issue with data sample

...
lyel's user avatar
  • 1
1 vote
1 answer
27 views

How to match the two datasets on Electoral bonds given by SBI?

Just stumbled across the Electoral bond dataset https://www.kaggle.com/datasets/shaundanielll/electoral-bond-data-state-bank-of-india There are two csvs- One contains Encashment data(Date of ...
dirtynerdy's user avatar
0 votes
1 answer
24 views

Combining datasets with different featues but, have similar names but, not exactly I can do name = name

I want to join datasets of data I have scraped from Open Critic, vgchartz, and Steam. All of these datasets have different fields but, I want to be able to join the 3 datasets to make one dataset with ...
Christopher Kinoshita's user avatar
1 vote
1 answer
302 views

FutureWarning in pandas when working with Series

I'm coding a custom (but simple) dataset for training an MLP with PyTorch. Basically, my data are numeric vectors of two classes so the whole dataset is a ...
leapofFaith's user avatar
2 votes
1 answer
38 views

Effect of Sequential Data Quality Variation on Transformer Model Training: Seeking Insights and Experiences

I'm exploring the training efficiency of transformer models against the backdrop of data quality sequencing. Specifically, I ponder whether arranging unlabeled data by presumed quality affects ...
Lukas N.P. Egger's user avatar
0 votes
0 answers
13 views

Trip/Hazard/Fall Dataset

I'm working on a project that involves describing/detecting 3 main situations from an image. First, if there's a trip hazard or o potential situation that could result in a fall, for example someone ...
Ricardo Mezarina's user avatar
1 vote
0 answers
114 views

Generating quality synthetic tabular data - is it possible when one's dataset is extremely small?

I've got a dataset consisting of only 17 samples and 6 continuous features (all values in the dataset contain decimals, although 2 features exhibit categorical-ish behaviour). I'm looking at the ...
user23493275's user avatar
0 votes
1 answer
24 views

How to create a multi label text classification model for small dataset in production [closed]

I have a multi-label text classification dataset which is very small around 80Kb, I am only going to receive a small amount of data for training from my client. But it is expected to build a high ...
Aayesha Qureshi's user avatar
0 votes
2 answers
39 views

Difference between (None, 2000, 8) and (2000, 8)

Input: a tensorflow dataset with 17000 items: ...
Fabi's user avatar
  • 11
0 votes
0 answers
19 views

Differentiating between runs that start at different timestamps

I have a CSV file of the latitude and longitude coordinates of trains in the format [timestamp, runID, latitude, longitude]. The problem is that I have tracked ...
make nice rhombus's user avatar
0 votes
0 answers
9 views

Which dataset could be a good choice to train Environment Sound Classification model for user environment awareness while wearing earbus?

Which dataset could be a good choice to train an Environment Sound Classification model for the following use case: use the model in the earbuds/earphones to detect important sound events in the user'...
Danijel's user avatar
  • 173
0 votes
0 answers
11 views

Dataset with replicates: aggregation or not?

I am currently developing a neural network tailored to a regression problem using a synthetic dataset derived from an experimental simulation campaign. Given the stochastic nature of both the ...
Andrea Ferrari's user avatar
0 votes
1 answer
79 views

Outdated Transformers TextDataset class drops last block when text overlaps. Replace by datasets Dataset class as input of Trainer train_dataset?

Why I try to replace the transformers TextDataset class with datasets Dataset class I stumbled upon this when I tried to make ...
questionto42's user avatar
0 votes
0 answers
15 views

How to identiyfy issues in structure or missing values in netCDF-file that contains weather data?

during processing of weather data files (wind x,y 10m), i get the follwing error: height3' not present in all datasets and coords='different'. Either add 'height3' to datasets where it is missing or ...
otk's user avatar
  • 101
0 votes
0 answers
15 views

How to generate a netCDF weather forecast for a missing day based on yesterdays forecast?

as can be seen here met.no, the forcast is missing for 20210823. This forcast for each day is a 66 hour forecast, mening it will cover the next "missing" day. Not knowing much about netCDF ...
otk's user avatar
  • 101
0 votes
0 answers
16 views

Iterating more quickly, shorter feedback cycles, general workflow practices

(caveat: I've not written that much ML code yet) When I'm writing non-ML code I like to create short feedback loops to check the validity of my code. I use unit-tests or sometimes short helper tools (...
Niels Bom's user avatar
  • 101
0 votes
1 answer
26 views

Where can I get 5000+ classified images of zoo animals? [closed]

please help! We are college students doing this for a project. The project is using neural networks and want to build a model that takes in an input of a colored image of an animal and outputs the ...
user90061's user avatar
0 votes
1 answer
28 views

Binary Classification Dataset Distribution

I have a question regarding the distribution of my dataset. I have a binary classification task which I need to label. I now want to know how I should distribute the labels for optimal training / ...
Apatus's user avatar
  • 1
0 votes
0 answers
9 views

Estimating the Cost of Creating an Image Dataset like GQA and Relevant Sources

I am currently working on a project that requires an extensive image dataset, similar to the GQA dataset. I am in the early stages of planning and would like to get an estimate of the cost involved in ...
Patrick Amadeus Irawan's user avatar
0 votes
1 answer
23 views

What are these 'code' columns in this csv and what does Pandas pivot method do with it?

In a csv discussed in the book HoML (hands on machine learning), what do the values in capital letters represent? Are these some sort of code? Also, it seems the pandas.pivot() methods does something ...
Yorick Dantuma's user avatar
0 votes
0 answers
33 views

How can I identify coverage types in NFL games using Computer vision

I am currently working on a project that classifies coverage types from sports highlights using advanced computer vision techniques. Next Gen Stats effectively utilizes tracking data to identify ...
Shah Zeb's user avatar
1 vote
1 answer
65 views

Correct way to take a subset of a dataset?

I am attempting a binary classification problem (using Weka). My dataset has 100,000 rows, 14 attributes (1 output variable). It takes already too long just to open the dataset in excel so I just know ...
FlexMcMurphy's user avatar
0 votes
0 answers
20 views

Should I log transform the data if the dataset is small?

Suppose I have a dataset with only about 25 rows. All the relevant columns are all integers. The ranges of them go from 1 million to 100 million, and the distributions are all skewed right. In this ...
DSGuest's user avatar
0 votes
0 answers
10 views

How to make duplicates unique in a DataFrame robustly?

I have the following table. I was able to parse it and display it as a DataFrame with pandas in python I need to change the Ref for every value that is duplicate such that it becomes unique. For ...
bem22's user avatar
  • 101
0 votes
1 answer
22 views

How should I plot 2D data as 3D plot?

I got data (1503, 38697) (cell(datapoint), genes(features)), each elements has integer values (gene expression). so something like ...
containletters's user avatar

1
2 3 4 5
31