Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

296 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
4answers
393 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
4
votes
1answer
3k views

Memory error on using data generator in keras

I am using the following augmentations on dataset of size 9 GB: ...
4
votes
0answers
231 views

Are SAS Data Storage Options designed for Big Data?

I understand that SAS has some data storage options in the form of Default Storage tables or SAS datasets - in the SAS library. Multi-dimensional Storage cubes - in the OLAP Server. Third Party ...
4
votes
4answers
201 views

How can I weigh observations differently that were provided for a time horizon?

I have 623 observations with one continuous dependent variable and 13 independent variables (continuous, categorical, and ordinal), defined based on researcher experience and literature review. I ...
3
votes
1answer
57 views

Relation between an underlying function and the underlying probability distribition function of data

I heard and read a lot of times the following statements and got a lot of confusion over time. Statement 1: The goal of machine learning is to get a function from the given data Statement 2: ...
3
votes
0answers
112 views

How do I implement masking in TensorFlow eager execution?

I am training a stateful RNN on variable length sequences (optional: see my previous question for more details). I padded the sequences to a fixed length with the value -1. The when batches are ...
3
votes
1answer
50 views

features importance

Suppose I have dataset labeled with two classes such as healthy and unhealthy and I applied feature selection (features importance)on dataset. How can I know that features are important to which ...
3
votes
0answers
26 views

Sub-sampling so that sample statistics match population statistics

I want to investigate the impact of various testing strategies on a product. Let's say chairs. I start with 500 random chairs that I've picked up from garage/yard sales. They come in all shapes and ...
3
votes
0answers
39 views

How to filter this signal to get a heartbeat signal?

I am giving my first steps in data analysis, gathering/cleaning. To learn, I am trying to create a simple code that can detect heartbeats from color variations from the image coming from the camera ...
3
votes
1answer
1k views

What is the best way to read SQL dataset in to Tensorflow?

What is the best way to read SQL database in to Tensorflow? Currently, I am using Postgres on server and developed DL algorithm on Tensorflow using Jupyter Lab. How can I import data into Jupyter Lab ...
3
votes
0answers
703 views

Preparing ground truth labels for YOLO3

I want to train YOLO3 for a custom dataset that has raw labels in JSON format. Each bounding box in JSON is specified as [x1, y1, x2, y2]. So far, I have ...
3
votes
2answers
360 views

Genetic neural network to satisfy variable number of inputs and outputs

I have what I propose as a solution to my problem, however I haven't ever seen it mentioned in this way, so I worry that there is a valid reason not to do things this way. I have a dataset of > 100,...
3
votes
1answer
1k views

Should we use discrete or continuous input for decision trees

I have 2 datasets, a continuous dataset(75 datapoints and 14 variables) and a discretized dataset which was made by placing the continuous datasets into buckets. I have built a decision tree ...
3
votes
0answers
29 views

Merging 2 datawarehouses

Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data ...
3
votes
1answer
163 views

Shifted feature distribution across different datasets

I am trying to validate a classifier using two different training and testing datasets. The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. ...
2
votes
1answer
41 views

Could I add a one hot encoding to each feature representing “has data” versus “has no data”

I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and ...
2
votes
1answer
49 views

Given a machine learning algorithm, what is the minimum size of the training set for it?

I understand that the more data we have, the more reliable is our model trained on that data. I also understand that the more parameters a machine learning model has, the more training data it ...
2
votes
2answers
27 views

How to gather training data for simple voice commands?

I'm trying to build a machine learning model for recognizing simple voice commands like up, down, left, etc. On similar problems based on images, I'd just take the picture and assign a label to it. ...
2
votes
2answers
541 views

Looking for animals dataset for deep learning classification

Do you know any datasets that contain animals and their accurate classifications? I am looking for any dataset that categorizes animals. For example: a dataset with insects, with an image of an ...
2
votes
1answer
209 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
2
votes
1answer
86 views

How to Setup Multivariate Time Series Dataset for Classification

First post on StackExchange. I’m fairly new to ML, with about 1 year of experience so please pardon any ignorance or misuse of terms. I have a multivariate time series dataset where I would like to ...
2
votes
0answers
21 views

Fuzzily join two large sets of postal addresses

I have two tables of postal address information - the one is about 2 million records, the other roughly 40 million. They have quite bad quality, and also are not quite compatible with each other (...
2
votes
0answers
16 views

Where should we release our video dataset for activity recognition?

We have an activity recognition dataset made of tens of thousands video clips. Which are the alternatives to release it to the research community? Is there any public website? I saw for instance ...
2
votes
1answer
49 views

What is the best way to organize the datasets for my task?

I am working on a task — geolocation estimation of Twitter users by using tweets only. I collected tweets (and users) from more than 6000 people in Twitter. Each user is associated with a city. In ...
2
votes
1answer
88 views

Dataset where svm performance is significantly different from random forest

Is there a specific dataset where svm performs significantly better or worse than random forest? I know that the performance could depend on the dataset but is there a specific dataset?
2
votes
1answer
93 views

Reinforcement Learning on real time data over a web server

Question: is it possible to implement a reinforcement learning model over a NodeJS server? This server would be receiving binary forms of data (open /close; yes/no) in real time. The objective for ...
2
votes
2answers
6k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
2
votes
0answers
529 views

Keras' fit_generator() is not calling my generator

When I call Keras' fit_generator(), passing in a custom generator class I created, I see "Epoch 1/1" in the spew and that's all. It hangs right there, and the ...
2
votes
0answers
67 views

detecting anomaly from cdr data

I need to find anomalies from this. The term anomaly in my case is in which square id and in which time the total activity goes higher, how can I detect anomalies with the help of machine learning ...
2
votes
0answers
54 views

Data formatting or Data preparation

...
2
votes
0answers
41 views

Making inferences from incomplete data

I have data which have complete information. Each record has one class assigned. On production, I won't be able to get so many information from a user, so I want to create a model which will be able ...
2
votes
0answers
6 views

Sacle lines and GOF stats for Metric MDS?

Sirs I am new to Orange3 and am getting to grips with Metric MDS - I think the paired links is inovative Usually MDS plots have scales at the bottom and left side but there do not appear to be present ...
2
votes
1answer
62 views

What kind of plot will be helpful for total count group data ( data grouped by month and year )?

I have a group data frame like this : ...
2
votes
0answers
19 views

Quantile regression with inhomogeneous density of points

I am working on a dataset that can be represented this way: We can see that the lower values on the y-axis are increasing linearly along the x-axis. I want to estimate the coefficients of this ...
2
votes
0answers
17 views

Longitudinal study - conversion long to wide - implications

I'm doing a longitudinal study and trying to find out patterns before a defined event happens. For instance, we have data: ...
2
votes
0answers
53 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
2
votes
0answers
43 views

Can you have too uniform test data in a feedforward neural network?

I been playing around trying to implement my own feedforward neural network. To try it out I decided on an easy example. 3 inputs, 3 output. When you send in (1,0,0)...
2
votes
0answers
236 views

Keras or TensorFlow Examples for Working with Large Text Datasets (~10M Sentences)

I'm looking for any tutorials or examples for using either Keras or TensorFlow on large text datasets (at least a few million sentences). I've looked at some of the examples using the ...
2
votes
0answers
46 views

Are there deduplication algorithms that do not work on a metric space?

Recently I got interested in the process of data cleansing and specifically in record linkage. Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some ...
2
votes
0answers
52 views

Machine Learning over distributed operating system

I want to implement Distributed operating system mutual exclusion using Machine learning algorithms but I am not getting dataset anywhere to perform. Is there anyway i can get the required dataset? I ...
2
votes
0answers
209 views

global flight network (with number of pessengers per year) dataset

Is there such a global flight network dataset that gives you: flight route (the connection between each airports or cities by some flight company) the number of passengers (per year) taking each ...
2
votes
0answers
339 views

Where can I find a >1K dataset of annotated store receipt pictures/scans?

I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice: vendor date of purchase total amount (...
2
votes
0answers
104 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
2
votes
0answers
128 views

How to quantitatively compare two or more complex data sets

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response ...
2
votes
2answers
268 views

Feature Engineering

I have a dataframe with around 37,000 rows and 54 columns. Out of these 54 columns two columns namely 'user_id' and 'mail_id' are provided in avery creepy format as shown below: ...
2
votes
0answers
29 views

User ratings and conditional values

I have a set of user transactions with a service and I suspect that there is a strong correlation between how users rate the service and how likely they are to use the service again. I have chosen to ...
2
votes
0answers
122 views

Good Libraries or Software for Temporal Bayesian Network Structure Learning?

I would like to infer structures of BNs with edges within and between time slices by using data. I tried libpgm (http://pythonhosted.org/libpgm/) and found that 2TBN is ideal for my purpose, but ...
2
votes
0answers
48 views

How to interpret a dataset with a pseudo-logarithmic trend?

I am dealing with a scatterplot where I am trying to figure out the relationship between two variables, but I have so far failed to identify what the best fit could be. The presence of zero values ...
2
votes
0answers
117 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
0answers
299 views

Creating Data model for mahout recommendation engine

I am trying to build an item-item similarity matching recommendation engine with mahout. The data set is as in the following format ( attributes are in text not in numerals format ) ...