Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

300 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
3k views

Memory error on using data generator in keras

I am using the following augmentations on dataset of size 9 GB: ...
4
votes
0answers
238 views

Are SAS Data Storage Options designed for Big Data?

I understand that SAS has some data storage options in the form of Default Storage tables or SAS datasets - in the SAS library. Multi-dimensional Storage cubes - in the OLAP Server. Third Party ...
3
votes
0answers
16 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
3
votes
1answer
57 views

Relation between an underlying function and the underlying probability distribition function of data

I heard and read a lot of times the following statements and got a lot of confusion over time. Statement 1: The goal of machine learning is to get a function from the given data Statement 2: ...
3
votes
1answer
320 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
3
votes
0answers
140 views

How do I implement masking in TensorFlow eager execution?

I am training a stateful RNN on variable length sequences (optional: see my previous question for more details). I padded the sequences to a fixed length with the value -1. The when batches are ...
3
votes
1answer
56 views

features importance

Suppose I have dataset labeled with two classes such as healthy and unhealthy and I applied feature selection (features importance)on dataset. How can I know that features are important to which ...
3
votes
0answers
40 views

How to filter this signal to get a heartbeat signal?

I am giving my first steps in data analysis, gathering/cleaning. To learn, I am trying to create a simple code that can detect heartbeats from color variations from the image coming from the camera ...
3
votes
1answer
2k views

What is the best way to read SQL dataset in to Tensorflow?

What is the best way to read SQL database in to Tensorflow? Currently, I am using Postgres on server and developed DL algorithm on Tensorflow using Jupyter Lab. How can I import data into Jupyter Lab ...
3
votes
0answers
755 views

Preparing ground truth labels for YOLO3

I want to train YOLO3 for a custom dataset that has raw labels in JSON format. Each bounding box in JSON is specified as [x1, y1, x2, y2]. So far, I have ...
3
votes
2answers
374 views

Genetic neural network to satisfy variable number of inputs and outputs

I have what I propose as a solution to my problem, however I haven't ever seen it mentioned in this way, so I worry that there is a valid reason not to do things this way. I have a dataset of > 100,...
3
votes
0answers
347 views

Where can I find a >1K dataset of annotated store receipt pictures/scans?

I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice: vendor date of purchase total amount (...
3
votes
0answers
30 views

Merging 2 datawarehouses

Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data ...
2
votes
0answers
34 views

Fixed-Width vs Adaptive Binning

I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
2
votes
2answers
30 views

How to gather training data for simple voice commands?

I'm trying to build a machine learning model for recognizing simple voice commands like up, down, left, etc. On similar problems based on images, I'd just take the picture and assign a label to it. ...
2
votes
2answers
1k views

Looking for animals dataset for deep learning classification

Do you know any datasets that contain animals and their accurate classifications? I am looking for any dataset that categorizes animals. For example: a dataset with insects, with an image of an ...
2
votes
1answer
113 views

How to Setup Multivariate Time Series Dataset for Classification

First post on StackExchange. I’m fairly new to ML, with about 1 year of experience so please pardon any ignorance or misuse of terms. I have a multivariate time series dataset where I would like to ...
2
votes
0answers
23 views

Fuzzily join two large sets of postal addresses

I have two tables of postal address information - the one is about 2 million records, the other roughly 40 million. They have quite bad quality, and also are not quite compatible with each other (...
2
votes
0answers
16 views

Where should we release our video dataset for activity recognition?

We have an activity recognition dataset made of tens of thousands video clips. Which are the alternatives to release it to the research community? Is there any public website? I saw for instance ...
2
votes
1answer
67 views

What is the best way to organize the datasets for my task?

I am working on a task — geolocation estimation of Twitter users by using tweets only. I collected tweets (and users) from more than 6000 people in Twitter. Each user is associated with a city. In ...
2
votes
1answer
101 views

Dataset where svm performance is significantly different from random forest

Is there a specific dataset where svm performs significantly better or worse than random forest? I know that the performance could depend on the dataset but is there a specific dataset?
2
votes
1answer
101 views

Reinforcement Learning on real time data over a web server

Question: is it possible to implement a reinforcement learning model over a NodeJS server? This server would be receiving binary forms of data (open /close; yes/no) in real time. The objective for ...
2
votes
0answers
587 views

Keras' fit_generator() is not calling my generator

When I call Keras' fit_generator(), passing in a custom generator class I created, I see "Epoch 1/1" in the spew and that's all. It hangs right there, and the ...
2
votes
0answers
73 views

detecting anomaly from cdr data

I need to find anomalies from this. The term anomaly in my case is in which square id and in which time the total activity goes higher, how can I detect anomalies with the help of machine learning ...
2
votes
0answers
55 views

Data formatting or Data preparation

...
2
votes
0answers
41 views

Making inferences from incomplete data

I have data which have complete information. Each record has one class assigned. On production, I won't be able to get so many information from a user, so I want to create a model which will be able ...
2
votes
0answers
20 views

Longitudinal study - conversion long to wide - implications

I'm doing a longitudinal study and trying to find out patterns before a defined event happens. For instance, we have data: ...
2
votes
0answers
58 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
2
votes
0answers
44 views

Can you have too uniform test data in a feedforward neural network?

I been playing around trying to implement my own feedforward neural network. To try it out I decided on an easy example. 3 inputs, 3 output. When you send in (1,0,0)...
2
votes
0answers
243 views

Keras or TensorFlow Examples for Working with Large Text Datasets (~10M Sentences)

I'm looking for any tutorials or examples for using either Keras or TensorFlow on large text datasets (at least a few million sentences). I've looked at some of the examples using the ...
2
votes
0answers
47 views

Are there deduplication algorithms that do not work on a metric space?

Recently I got interested in the process of data cleansing and specifically in record linkage. Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some ...
2
votes
0answers
52 views

Machine Learning over distributed operating system

I want to implement Distributed operating system mutual exclusion using Machine learning algorithms but I am not getting dataset anywhere to perform. Is there anyway i can get the required dataset? I ...
2
votes
0answers
225 views

global flight network (with number of pessengers per year) dataset

Is there such a global flight network dataset that gives you: flight route (the connection between each airports or cities by some flight company) the number of passengers (per year) taking each ...
2
votes
0answers
106 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
2
votes
0answers
129 views

How to quantitatively compare two or more complex data sets

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response ...
2
votes
0answers
29 views

User ratings and conditional values

I have a set of user transactions with a service and I suspect that there is a strong correlation between how users rate the service and how likely they are to use the service again. I have chosen to ...
2
votes
0answers
122 views

Good Libraries or Software for Temporal Bayesian Network Structure Learning?

I would like to infer structures of BNs with edges within and between time slices by using data. I tried libpgm (http://pythonhosted.org/libpgm/) and found that 2TBN is ideal for my purpose, but ...
2
votes
0answers
49 views

How to interpret a dataset with a pseudo-logarithmic trend?

I am dealing with a scatterplot where I am trying to figure out the relationship between two variables, but I have so far failed to identify what the best fit could be. The presence of zero values ...
2
votes
0answers
126 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
0answers
83 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
2
votes
0answers
303 views

Creating Data model for mahout recommendation engine

I am trying to build an item-item similarity matching recommendation engine with mahout. The data set is as in the following format ( attributes are in text not in numerals format ) ...
2
votes
1answer
24 views

is it a good idea to take the derivative or integral of some features and add them as new features in machine learning?

I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column ...
2
votes
1answer
25 views

How to make smaller categories with factor character variables

I have this data set with consist of ISO3166 Alpha-2 codes for countries. Example: DE, AD, AE etc They are coded as factor variables in R and there are about 173 observations. Now because there are ...
2
votes
2answers
55 views

Why would 2 sets of similar training samples take significantly longer to train?

I've built a fully-connected feed-forward neural network to recognize handwritten digits. I used MNIST and another very similar dataset (containing Arabic digits - same training set and test set count ...
1
vote
1answer
13 views

Creating pronunciation dictionary for ASR

I am working on ASR(automatic speech recoginition) on Somali data as master thesis and now I am stuck with how to create a phonetics or pronunciation dictionary for it. I searched over net and could ...
1
vote
0answers
9 views

Non-parametric regression on set of time series: One model for each or one for all series?

Let's say I have a set of 1D time series which values have been samples in equip-distant time steps with timestamps $1,2,3,...$, they have all the same lengths and are somewhat similar in shape. I ...
1
vote
0answers
14 views

Tool/Technology/Framework to process file with 1GB

I'm new to datascience, but I have the following problem: 1) Read a file of 1GB, which each line is a json object 2) There are two more files, much smaller, which I need to JOIN some data In this ...
1
vote
0answers
26 views

How to estimate and/or determine how much data is enough to train a model?

To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of ...
1
vote
1answer
35 views

Autoencoder anamoly detection

I recently learnt about the anamoly detection using autoencoders(specifically denoisinng autoencoders).To train the autoencoders do we need a data having some pattern? or is there some way to ...
1
vote
0answers
51 views

Alignment of time series

I have two time series of different lengths: ...

1 2 3 4 5 6