Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Incompatible shape error when trying to fit my model

I am using keras to make a CNN model. I have a dataset of images which consists of training and validation tfrecord files, the images in the tfrecords are grayscale. I load in the my 2 datasets by ...
0
votes
1answer
19 views

Accuracy on Validation and Test set, Overfit?

Just a quick question, I am building a ML model right now however I am receiving very similar (72.2 and 72.4 for example)% for both Accuracy and F1-Score on my Validation Dataset and my unseen Test ...
1
vote
1answer
32 views

Call routing using AI

I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can ...
0
votes
1answer
20 views

KNN Variance using a high value of K and cross-validation

it has come to my understanding, that a value of K=1, gives a high variance because we are only using only one data point, hence we are very likely to model the noise in that training example. Bias: ...
0
votes
1answer
319 views

Sampling labeled data for anomaly detection

I'm currently working on a project that requires the use of unsupervised anomaly detection, but I'm unable to find a relevant data set, so I'm considering the following option: Assuming I have a data ...
0
votes
0answers
19 views

Why don't I find a decision making problem and dataset [closed]

I am a master student trying to work on decision support systems and improving their accuracy with machine learning techniques, namely neural nets for now. For 2 months, I have been trying to start to ...
1
vote
1answer
2k views

Extracting data and saving data from hdf5

I have a hdf5 file, and I want to extract a part of the data and save it as the same format. The data type is DATATYPE H5T_IEEE_F32BE So far I have ...
0
votes
1answer
957 views

Should the type of Boolean categorical features be numerical or categorical after encoding?

There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1. I will apply correalation elimination on ...
1
vote
1answer
31 views

How to generate training instances from multivariate normal distribution?

I would like to generate a training and test set for the binary classification problem. I want the instances to be generated from a multivariate normal distribution and have 5 features. Can someone ...
0
votes
0answers
12 views

How to resize image along with their mask?

I have original images of the size 1935x1481. I am using labelme to annotate the images. I am creating polygons on the original image. Is there a way to resize the image along with their mask? I am ...
0
votes
0answers
11 views

Grape dataset for smart viticulture

I'm involved in a smart viticulture project. I need some datasets for machine learning models. Datasets about grape disease or ripeness related to climate variables or grape.
0
votes
1answer
19 views

Currency Normalization for Salary Prediction

I have a dataset (350k data points) with data of employees across different regions over the last 10 years. The dataset consists of their skills, the region they are in, the industry, their current ...
2
votes
0answers
15 views

Best Method for Using Experts to Score Qualitative Data

TL;DR: Looking for recommendations or resources on the best method to have subject matter experts score qualitative data to train ML model. Problem: I am working on a problem in the biochemistry ...
1
vote
1answer
21 views

What if irrelevant features impact the outputs?

We know that weather conditions in China is not very useful to predict price of a house in Spain (as life experience). However, when we drop that China's weather condition, the accuracy is reduced ...
0
votes
0answers
6 views

High resolution satellite data for climate analysis

i'm into a smart agriculture project. I need to get past info about climate in a specific region. Could you recommend satellite climate data with highest resolution?
0
votes
1answer
71 views

How to write custom de-identification algorithm in Python?

I have tried a simple algorithm to anonymize my data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values. The data ...
8
votes
3answers
5k views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
0
votes
0answers
128 views

Does Call Detail Record(CDR) data record different cell-site/tower on incoming and outgoing call/text?

I am working on CDR data to find important locations of user. The data I have does not have coordinates so I will be using cell-site location as user's location. But after doing some working on data I ...
0
votes
1answer
340 views

LSTM - How to prepare train from a dataset which contains multiple observations for different events

I m using LSTM in a project related to MobiFall dataset which contains falls and daily activitives - such as walking, sitting etc - sensed by accelerometer, gyroscope and orientation sensors in x,y,z ...
1
vote
1answer
50 views

Algorithm for user profiling without distinct profiles

I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide ...
0
votes
0answers
25 views

Examples of Open source huge business intelligence datasets

I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
1
vote
1answer
34 views

Application of Machine learning in modelling experimental system as a PhD topic

I am working as a control Engineer in a high energy physics lab. I am looking for a PhD thesis topic. Our experimental system is highly non linear and can't be modeled using standard white box ...
0
votes
2answers
99 views

Public dataset for account to account payments

I'm looking for a dataset which contains account to account payments (bank transfers). Ideally, this dataset would contain labeled data for transactions or accounts known to be victims of phishing ...
1
vote
1answer
126 views

How to prepare Audio-text data for speech recognition

I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I ...
1
vote
0answers
28 views

Is scraping job posts and gathering skills tags illegal? [closed]

I want to scrape a freelancer site (freelancer.com) and gather their public projects skill tags. Is this illegal to do? And is publishing it illegal?
1
vote
2answers
34 views

Converting data format

I'm trying to use the recent COVID-19 data from the site of Italian Civil Protection, but they use a rather complicated time format that I'm finding troublesome as a novice to plot as data in a graph. ...
0
votes
1answer
21 views

Which dataset I should use when I am retraining my model?

I trained my deep learning model using x dataset and now I got new dataset and I give it name as y. I want to retrain my model on this new dataset which is y. Do I need to use x+y dataset or just y? ...
3
votes
1answer
1k views

How to load numerous files from google drive into colab

I am trying to load in 30k images (600mb) from Google drive into Google Colaboratory to further process them with Keras/PyTorch. Therefore I have first mounted my Google drive using: ...
6
votes
3answers
1k views

What are bias and variance in machine learning?

I am studying machine learning, and I have encountered the concept of bias and variance. I am a university student and in the slides of my professor, the bias is defined as: $bias = E[error_s(h)]-...
0
votes
0answers
21 views

dataframe merge on string not working

having two dataframes each having common column which is country_code. while merging it by 'inner' join, it's returns empty dataframe. but i can see there is many values are common like IND, USA.... <...
2
votes
2answers
569 views

How to download BAIR action free robot pushing dataset?

I'm trying to download BAIR action free robot pushing dataset. I tried downloading from here. In browser, it shows its size is 30GB, but downloads some data and ...
1
vote
1answer
103 views

How to train on extended data set correctly

I have trained my classifier on pictures with a mixture of several classes on each picture, e.g. A-F. The classifier is able to (nearly) correctly segment those classes on the images. Now I got more ...
0
votes
0answers
11 views

Prediction ability by small sample size datasets

All the datasets i have for modelling are relativ small - ca. 80 rows and there is no way how to increase the sample size. If there is no significant or dominant parameter / input (usually there is ...
0
votes
1answer
129 views

The distribution of dataset train and test are the differents, how to fix this?

I am new in data science and like some help to understand my problem. For instance, I have two signals non-stationary for the same condition (figure 1). I acquisition them at different times(in the ...
0
votes
1answer
68 views

Comparing Dataset - Should I use the same Test dataset?

I am training ML CNN model. I want to compare different images dataset. The dataset all have different characteristics (Translated or not, Rotated or not, etc.). I do not modify the ML model between ...
2
votes
1answer
56 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
0
votes
0answers
11 views

Computing class_weights using native tf.Dataset

There are already some answers on DSE about computing class-weights here but they require you to already know the number of samples in each class and assume it is ...
2
votes
3answers
76 views

What is the best way for synthetic data generation while maintaining privacy?

For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to ...
1
vote
1answer
49 views

Unable to understand which features to choose

I am a newbie here, but I am trying to work with a dataset which gives the attempt at the goal by a footballer,which will predict one of 2 possible outputs - whether or not they could score the goal ...
0
votes
1answer
292 views

How do I calculate a similarity matrix with a Student-t kernel?

As the title says, how do I calculate a similarity matrix with an un-normalized Student-t kernel? I'm attempting to calculate Kullback-Leibler divergence for different t-SNE runs, but need a Q-matrix ...
2
votes
1answer
218 views

What is the best way to organize the datasets for my task?

I am working on a task — geolocation estimation of Twitter users by using tweets only. I collected tweets (and users) from more than 6000 people in Twitter. Each user is associated with a city. In ...
2
votes
2answers
56 views

How to gather training data for simple voice commands?

I'm trying to build a machine learning model for recognizing simple voice commands like up, down, left, etc. On similar problems based on images, I'd just take the picture and assign a label to it. ...
0
votes
0answers
6 views

How to perform cropping on large file dataset using ImageGenerators?

I have a large dataset of RGB images with 220x220x3. I want to crop the region of interest from each of them. For that, I have defined my own function. Im trying to use this with DataImageGenerators ...
1
vote
1answer
43 views

How to measure retention statistics?

I have a dataset with the ID, name, joining date, leaving date as features. I was asked to measure employee retention and health of it. What can I derive from these? What are some latest trends and ...
1
vote
1answer
19 views

Pandas: Group by Single Column Entries [closed]

So have this table above. I'm trying to aggregate the occupations such that the table results in: I've tried using df.groupby(['Occupation']) but I get an error. ...
8
votes
4answers
7k views

One Hot encoding for large number of values

How do we use one hot encoding if the number of values which a categorical variable can take is large ? In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
1
vote
1answer
715 views

LFR as synthesized data for dynamic community detection methods

In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper ...
1
vote
1answer
117 views

Dataset of extremely low-dimensional images for PCA

I am looking for a public data-set of images that differ from each other only slightly, so that after applying PCA they can be reconstructed with a small error from ...
2
votes
2answers
62 views

looking for public dataset for stock market

I want to do some modelling and data visualization on historical stock data, including price, volume, financials, etc. Is there an public dataset available for stock price history? I looked at a few, ...
1
vote
1answer
121 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...

1
2 3 4 5
24