Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

How to do parameter estimation on relatively few samples?

Context: I have a time series on which I've trained an autoencoder for anomaly detection (AD) purposes. My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one week ...
0
votes
1answer
32 views

How to split data into training, validation, test data sets if the data is non-stationary?

When splitting data into training, validation, test data sets to be fed to machine learning model, the data is ideally expected to be stationary. However, in the real world, some data is non-...
1
vote
2answers
757 views

training neural net with multiple sets of time-series data

I have the following data ($x^1_i$, $y^1_i$) for $i=1,2,...N_1$ ($x^2_i$, $y^2_i$) for $i=1,2,...N_2$ ... ($x^m_i$, $y^m_i$) for $i=1,2,...N_m$ Is it possible to train a neural net to produce ...
28
votes
8answers
5k views

Why is it wrong to train and test a model on the same dataset?

What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?
1
vote
1answer
114 views

Dataset of extremely low-dimensional images for PCA

I am looking for a public data-set of images that differ from each other only slightly, so that after applying PCA they can be reconstructed with a small error from ...
1
vote
1answer
29 views

Call routing using AI

I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can ...
0
votes
1answer
113 views

The distribution of dataset train and test are the differents, how to fix this?

I am new in data science and like some help to understand my problem. For instance, I have two signals non-stationary for the same condition (figure 1). I acquisition them at different times(in the ...
3
votes
1answer
3k views

Reinforcement Learning on data only (NO emulators)

My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...). It is too costly for us ...
0
votes
0answers
11 views

Forecasting: Regression - Worse performance for a subset of data

I have a dataset for which I'm using different forecasting methods (Ridge/Lasso Regression / Random Forests / AdaBoost / Gradient Boosting) to compare their performance. For the full dataset, I ...
0
votes
0answers
10 views

In a dataset, how to detect simple relationship between two columns (datatype not restricted to numerical values)?

Suppose, I have a dataset with the following features: (FirstName, LastName, FullName, SchoolJoiningDate, SchoolGraduationDate) My aim is to validate the data before feeding it into the ML pipeline. ...
-2
votes
1answer
400 views

Training a model to predict duplicate records in machine learning

I have the adjustment data in telecom domain: There are problems that sometimes the automation script generates the bill twice which causes duplicate records. I want to know how to prepare the data ...
0
votes
0answers
11 views

query category wise dara

I have stored the some products and there respective category in sql. Like this. Now I want fetch all product from table with respective category. output should be=>
3
votes
1answer
67 views

Estimating and/or determining the amount of data sufficient enough to train a model

To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of ...
0
votes
1answer
63 views

How can I generate reasonable dummy/artificial data from a pre-existing time-series data?

I have a dataset like this, basically all numerical time-series data. I would like to generate dummy/artificial/fake data for future values of this, preferably in python. How can I achieve this for ...
2
votes
1answer
213 views

What is the best way to organize the datasets for my task?

I am working on a task — geolocation estimation of Twitter users by using tweets only. I collected tweets (and users) from more than 6000 people in Twitter. Each user is associated with a city. In ...
0
votes
0answers
14 views

Same accuracy on KNN for different distance metrics

I am new to data analysis and I am trying to run a kNN classifier on a lung cancer dataset with multiple attributes. For all k values I tasted (1 to 10), I obtain the same accuracy when using either ...
2
votes
0answers
41 views

Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.). It is too costly for us to emulate agents. We ...
1
vote
0answers
24 views

Non-uniform class occurances in input data for classification task - how to tackle it?

So, I gathered political articles for my thesis, now I want to be able to classify given text. Though the classes distribution is actually crazy. Class 1: 964 docs Class 2: 37,020 Class 3: 640 Class ...
1
vote
1answer
543 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
1
vote
1answer
82 views

Split Dataset in two Explainable Subsets

is there a general approach for splitting a dataset into two or more subsets so that a classification problem can be solved with higher accuracy for the resulting separate subsets? I have a very ...
9
votes
6answers
791 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
1
vote
0answers
17 views

Making a tagged Part-of-Speech corpus with the help of a lexicon

I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe. Also, I have a list of tokens (words) in another Dataframe. I want to take each ...
0
votes
0answers
45 views

How to predict variables based on multiple samples?

The problem I tried to do some ML models but everytime I had some weird plots and I couldn't understand these scores. I'm relatively new to ML and maybe someone can help me with this data with some ...
0
votes
0answers
10 views

Creating the Yolo Training datasets from Scrapped video as training model

I have gathered the 200 car type video for one car type and model Auto detection with YOLO (Accessible by this link), and converted the video of folders to photo by this colab page shown here: So i ...
-1
votes
1answer
61 views

Can we identify that an academic dataset was used for commercial purpose [closed]

There are many datasets released on the internet. Authors of many of these datasets state that the datasets are strictly for academic usage and not for commercial purposes. Although some datasets are ...
0
votes
1answer
23 views

How to merge all the data to have a final dataset [closed]

I am working with a problem that has different tables, end goal to predict if a customer will end up subscribing based on purchases. Mother table containing user_id,register_reason ...
0
votes
0answers
43 views

How to set thresholds to automate the management of model drift?

I have a trained ML Model and new data coming in every week. The new data sometimes vary too much (in a statistical sense) and, therefore, the performance of the ML model degrades. The data-set schema ...
0
votes
1answer
29 views

Encoding of high cardinality multi-label categorical feature?

This is the problem of binary classification: "1" - the subscriber is a driver (belongs to the segment of drivers), "0" - the subscriber is not a driver (does not belong to the ...
2
votes
2answers
74 views

RMSE is higher for bigger values of target variable - how to decrease

I am solving a problem with machine learning and I have some data with two integer type independent variables and a continuous dependent variable. I am optimising to RMSE. I had fairly large RMSE ...
8
votes
3answers
6k views

Where can I find freely available multi-label datasets online? [closed]

I'm trying to find multi-label classfication datasets, which are available for free online. By "multi-label" I mean that each instance can be labeled with anywhere from a single to $k$ labels, where ...
0
votes
0answers
32 views

NLP - Found input variables with inconsistent numbers samples

I'm trying to train a model to read the greetings from the sample dataset collected from Tripadvisor and I've been getting the following error when I am trying to ...
2
votes
2answers
2k views

Train-Test split for a recommender system

In all implementations of recommender systems I've seen so far, the train-test split is performed in this manner: ...
0
votes
1answer
37 views

Augmentation on test dataset and validation dataset

I'm training a segmentation model (computer-vision). Thus, my dataset contains images and masks (binary segmentation of objects). I'm augmenting the training dataset (applying random crop, rotation or ...
0
votes
2answers
621 views

Classification Dataset which is linearly non separable

I am trying to find a dataset which is linearly non-separable. I checked the Iris dataset and the UCI website says: The data set contains 3 classes of 50 instances each, where each class refers to ...
4
votes
2answers
171 views

Was this dataset analysed correctly? [closed]

There's a Twitter thread going around that claims there are signs of voter fraud due to anomalies in the election vote count data set. You can download the dataset here and find the script to generate ...
1
vote
1answer
18 views

Negative examples and false positives in object detection using SSD model

I am training a model to detect some objects. I have 8 classes and there are other similar objects that I don't want the model to detect. I have tried to make an unknown class for them but they are so ...
2
votes
1answer
167 views

How does skewed data affect deep neural networks?

I'm playing around with deep neural networks for a regression problem. The dataset I have is skewed right and for a linear regression model, I would typically perform a log transform. Should I be ...
0
votes
0answers
15 views

Data storage for Intrusion Detection System

I write an IDS(in C) for a KNX network for my thesis. For this, I just store all the telegrams within an SQLite database. But I'm not sure if a SQL database is the best here. I want to "ask ...
1
vote
2answers
26 views

How to work with Sensitive (personal) Data, without actually seeing it?

I'm currently working on a project that has sensitive data. I'm allowed to see the data on a single platform (company PC), but outside of that platform I cannot process it. I'd like to train a ML ...
-1
votes
1answer
37 views

Problem importing imblearn [closed]

I am trying to do: import imblearn in my jupyter notebook, but when I run the cell I get the following error message: ...
0
votes
0answers
17 views

How to correctly label images with multiple objects

I have 3 types of images: A: Images of apples B: Images of bananas, however some of these banana images also contain apples in the observable background Is it enough to just label the bounding boxes ...
0
votes
1answer
28 views

Training Accuracy is getting higher, but Valid Loss and Accuracy is same every epoch

I'm doing a transfer learning with ResNet50. My dataset is clothes(224x224x3), and 49 category(classes) -> training data 1000 per 1 category, total 49000. And valid data 200 per 1 category, total ...
5
votes
2answers
425 views

Can all data be represented tabularly?

... assuming that tables allowed for nested non-primitive values. I'm trying to think of any instance where data cannot be represented as tables (or a series of tables)... and I can't think of any. ...
3
votes
2answers
195 views

Training, Testing and Validation Dataset [closed]

I'm training a Unet model for tumor segmentation. I have a dataset of 400 patients for that. The used images are CT scans (3D images) that I divide into 2D images (for a total of 30k 2D images). I am ...
1
vote
0answers
17 views

Inferring composition of a subset of classes of multiclass dataset

I've come across a problem in assessing the design of a Machine Learning solution I am involved in, which I will describe using a commonly known dataset, the Iris Dataset, and some drawings. Suppose ...
3
votes
1answer
250 views

German Chatbot or conversational AI

I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set! So does it make sense to use google translator API to translate the ...
0
votes
1answer
29 views

Reuters Dataset Labels description

the list of Reuters dataset labels are provided with the dataset and also available in varous online resources. Here is an example. But I couldn't find what each of these labels actually means. I was ...
0
votes
1answer
41 views

How to normalize the data correctly in spam dataset

I'm working on the spam dataset to classify the inputs into binary classes. my problem is that: the observations in the dataset are floats small numbers in the first 53 column, and the 54 is float ...
-1
votes
2answers
2k views

How to deal with nan values after merging / joining two dataframes? [closed]

A lot of time after merging two pandas dataframes, I end up with NaNs in the new dataframe, that's just how the way it is, because one csv does not have all the ID's that the other has (Two dataframes ...
5
votes
3answers
101 views

Loading collections of datasets - Python code examples

Sometimes you might want to check your ideas on multiple datasets. There are several places with datasets collections. Question: Please share some Python scripts how to download multiple datasets from ...

1
4 5
6
7 8
24