Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
2
votes
1answer
58 views

Are there deduplication algorithms that do not work on a metric space?

Recently I got interested in the process of data cleansing and specifically in record linkage. Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some ...
1
vote
1answer
45 views

Unable to understand which features to choose

I am a newbie here, but I am trying to work with a dataset which gives the attempt at the goal by a footballer,which will predict one of 2 possible outputs - whether or not they could score the goal ...
3
votes
1answer
783 views

How to load numerous files from google drive into colab

I am trying to load in 30k images (600mb) from Google drive into Google Colaboratory to further process them with Keras/PyTorch. Therefore I have first mounted my Google drive using: ...
2
votes
2answers
47 views

How to gather training data for simple voice commands?

I'm trying to build a machine learning model for recognizing simple voice commands like up, down, left, etc. On similar problems based on images, I'd just take the picture and assign a label to it. ...
0
votes
0answers
11 views

why does tox21 loads as having 12 classes in pytorch geometric?

Why is the tox21 dataset loaded with predefined 12 classes when it should be binary classification, thus two classes? Code sampled modified: ...
0
votes
2answers
29 views

Converting data format

I'm trying to use the recent COVID-19 data from the site of Italian Civil Protection, but they use a rather complicated time format that I'm finding troublesome as a novice to plot as data in a graph. ...
0
votes
1answer
281 views

How do I calculate a similarity matrix with a Student-t kernel?

As the title says, how do I calculate a similarity matrix with an un-normalized Student-t kernel? I'm attempting to calculate Kullback-Leibler divergence for different t-SNE runs, but need a Q-matrix ...
0
votes
2answers
322 views

Splitting data in scikit-learn

I know how to split the dataset into train and test sets using train_test_split but is there any way that I can split the dataset into three different sets, i.e., &...
1
vote
1answer
14 views

What is the meaning of pct_trend_adjusted in FiveThirtyEight's polling data?

I'm working on polling data from fivethirtyeight. What is the meaning of the pct_trend_adjusted column here? I understand that pct stands for percentage, but I don'...
2
votes
2answers
290 views

Feature Engineering

I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below: ...
0
votes
1answer
47 views

Comparing Dataset - Should I use the same Test dataset?

I am training ML CNN model. I want to compare different images dataset. The dataset all have different characteristics (Translated or not, Rotated or not, etc.). I do not modify the ML model between ...
2
votes
2answers
2k views

How to download images and bounding boxes from imageNet?

I am doing object detection for a specific class, say, chairs . I want to download images of chairs from imageNet. I also want to download the annotation xml files ...
5
votes
2answers
780 views

What are bias and variance in machine learning?

I am studying machine learning, and I have encountered the concept of bias and variance. I am a university student and in the slides of my professor, the bias is defined as: $bias = E[error_s(h)]-...
1
vote
1answer
710 views

LFR as synthesized data for dynamic community detection methods

In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper ...
1
vote
1answer
19 views

Is it better to augment both training and validation sets or just training set?

Is it better to augment data both training and validation sets or just the training set in order to achieve the best accuracy possible on a convolutional neural network? why?
0
votes
1answer
6k views

How to find 4th point equidistant from 3 other points

For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The ...
2
votes
3answers
66 views

What is the best way for synthetic data generation while maintaining privacy?

For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to ...
0
votes
0answers
16 views

Is there a pre-trained network trained on RGB-D (4) channels?

The most used pre-trained networks for computer vision (e.g. ResNet50) are trained on 3 channels (RGB). At the same time, many cameras used in robotics return RGB-D outputs, that is including depth ...
3
votes
2answers
428 views

Genetic neural network to satisfy variable number of inputs and outputs

I have what I propose as a solution to my problem, however I haven't ever seen it mentioned in this way, so I worry that there is a valid reason not to do things this way. I have a dataset of > 100,...
1
vote
5answers
218 views

Identifying similar data points

I have a data set where customers who are non-delinquent with us have defaulted with others. So from the total data set, 30% are those who defaulted with others but 70% are those who are good with us ...
1
vote
1answer
24 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
1
vote
2answers
36 views

Negative Value in Covid-19 Dataset

I am currently practicing my data visualization skills with ourworldindata dataset. I noticed there is a negative value of new_cases in Benin country. Out of ...
0
votes
0answers
16 views

How to do parameter estimation on relatively few samples?

Context: I have a time series on which I've trained an autoencoder for anomaly detection (AD) purposes. My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one week ...
0
votes
1answer
25 views

How to split data into training, validation, test data sets if the data is non-stationary?

When splitting data into training, validation, test data sets to be fed to machine learning model, the data is ideally expected to be stationary. However, in the real world, some data is non-...
0
votes
1answer
30 views

How do we choose which feature(column) should we normalize or scale

I am a newcomer to machine learning, I was going through scaling and normalization. When I tried to explore most of the documentation on web, I found people explaining that the dataset needs to be ...
1
vote
2answers
747 views

training neural net with multiple sets of time-series data

I have the following data ($x^1_i$, $y^1_i$) for $i=1,2,...N_1$ ($x^2_i$, $y^2_i$) for $i=1,2,...N_2$ ... ($x^m_i$, $y^m_i$) for $i=1,2,...N_m$ Is it possible to train a neural net to produce ...
28
votes
8answers
5k views

Why is it wrong to train and test a model on the same dataset?

What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?
1
vote
1answer
114 views

Dataset of extremely low-dimensional images for PCA

I am looking for a public data-set of images that differ from each other only slightly, so that after applying PCA they can be reconstructed with a small error from ...
1
vote
1answer
29 views

Call routing using AI

I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can ...
0
votes
1answer
96 views

The distribution of dataset train and test are the differents, how to fix this?

I am new in data science and like some help to understand my problem. For instance, I have two signals non-stationary for the same condition (figure 1). I acquisition them at different times(in the ...
3
votes
1answer
3k views

Reinforcement Learning on data only (NO emulators)

My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...). It is too costly for us ...
0
votes
0answers
11 views

Forecasting: Regression - Worse performance for a subset of data

I have a dataset for which I'm using different forecasting methods (Ridge/Lasso Regression / Random Forests / AdaBoost / Gradient Boosting) to compare their performance. For the full dataset, I ...
1
vote
1answer
98 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
0
votes
0answers
9 views

In a dataset, how to detect simple relationship between two columns (datatype not restricted to numerical values)?

Suppose, I have a dataset with the following features: (FirstName, LastName, FullName, SchoolJoiningDate, SchoolGraduationDate) My aim is to validate the data before feeding it into the ML pipeline. ...
-2
votes
1answer
396 views

Training a model to predict duplicate records in machine learning

I have the adjustment data in telecom domain: There are problems that sometimes the automation script generates the bill twice which causes duplicate records. I want to know how to prepare the data ...
0
votes
1answer
36 views

How should I sample my validation set if I randomly sample training data?

I have: training dataset of size 150k. validation dataset of size 19k. At each epoch I randomly sample without replacement 10k datapoints for training because I get Out of Mem Errors. I need to ...
0
votes
0answers
11 views

query category wise dara

I have stored the some products and there respective category in sql. Like this. Now I want fetch all product from table with respective category. output should be=>
0
votes
1answer
68 views

Sklearn SVM question classification

So, I have found that there are many ways to classify words with sklearn's SVM algorithm. But I want to classify questions by taxonomy, as shown in the following dataset: The goal of this task is to ...
3
votes
1answer
62 views

Estimating and/or determining the amount of data sufficient enough to train a model

To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of ...
0
votes
1answer
31 views

How can I generate reasonable dummy/artificial data from a pre-existing time-series data?

I have a dataset like this, basically all numerical time-series data. I would like to generate dummy/artificial/fake data for future values of this, preferably in python. How can I achieve this for ...
0
votes
1answer
19 views

Log scaling whole dataset

I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value. I am wondering how I can get rid of this -infinity? I have been advised to add a small value ...
2
votes
1answer
198 views

What is the best way to organize the datasets for my task?

I am working on a task — geolocation estimation of Twitter users by using tweets only. I collected tweets (and users) from more than 6000 people in Twitter. Each user is associated with a city. In ...
0
votes
0answers
14 views

Same accuracy on KNN for different distance metrics

I am new to data analysis and I am trying to run a kNN classifier on a lung cancer dataset with multiple attributes. For all k values I tasted (1 to 10), I obtain the same accuracy when using either ...
0
votes
1answer
23 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
2
votes
0answers
35 views

Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.). It is too costly for us to emulate agents. We ...
1
vote
0answers
19 views

Non-uniform class occurances in input data for classification task - how to tackle it?

So, I gathered political articles for my thesis, now I want to be able to classify given text. Though the classes distribution is actually crazy. Class 1: 964 docs Class 2: 37,020 Class 3: 640 Class ...
1
vote
2answers
2k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
14
votes
4answers
20k views

One hot encoding alternatives for large categorical values

I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns? I found this interesting link. But they are ...
0
votes
1answer
143 views

Comparing data sets with different measurements

I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis. One piece of data (https://www.pandasecurity....
1
vote
1answer
506 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...

1 2 3
4
5
23