Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
-1
votes
1answer
162 views

Synonym handling replacement issue with UDF in Apache Spark

We are currently working to replace the synonyms of certain words in a Row. We are facing the problem as shown below. ...
1
vote
1answer
81 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
2answers
3k views

What exactly is a skewed dataset ?

Like above, I'd like to know, what exactly a skewed dataset is. The explanation from stats.se.com sounds to me more like what I call an imbalanced dataset? What is the distinction?
2
votes
4answers
161 views

I have n dimensional data and I want to check integrity, can I downgrade to 2 dimensional feature space via PCA and do so?

Say I have n dimensional data samples. I want to check the integrity of the features, if they are good representation of the respective classes, i.e. these features are good or not. My plan is: I ...
3
votes
2answers
635 views

Predicting hardware failures with limited data

I am exploring using machine learning to predict if a particular hardware component would fail within a timeframe, say 3 months. The ultimate goal is to minimize physical human inspection so that ...
3
votes
0answers
345 views

Where can I find a >1K dataset of annotated store receipt pictures/scans?

I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice: vendor date of purchase total amount (...
0
votes
1answer
79 views

City pollution data for data mining project [closed]

I want to do my MSc Project on Data Mining. And I want to work on pollution in cities. Are there accessible and useable pollution data (any type of pollution. e.g. air, water or forest pollution) in ...
3
votes
2answers
63 views

I want to add demographic data to a data set. Any suggestions on where to find zip code level data?

Ideally, I would like to add demographic features to my data. I have zip codes for each observation, so I was hoping to add in the demographic data by zip code. Unfortunately, I cannot find any ...
0
votes
1answer
66 views

How should I expose data from my app for data scientists?

I'm the product manager for an online app. I'm currently researching a new feature where our users will be able to access "all their raw application data". This data is likely to be used by data ...
2
votes
1answer
3k views

How to treat outliers in a time series dataset?

I've read the following article about how to treat outliers in a dataset: http://napitupulu-jon.appspot.com/posts/outliers-ud120.html Basically, he removes all the y which has a huge difference with ...
7
votes
5answers
444 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
1
vote
1answer
881 views

How do I represent a hidden markov model in data structure?

My task involves a POS Tagging using HMM. I am given a training data set (word/tag). I have to write a file with transition probabilities and emission probabilities. I am currently using a nested ...
4
votes
1answer
24k views

Having trouble installing and loading tidyverse- No DIB package [closed]

after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse. I get the following message, seemingly because the package DBI is missing: ...
1
vote
1answer
528 views

GANs to augment training data

I have been reading about Generative Adversarial Networks (GANs) and was wondering if it would make sense to train a generator function only to use it for creating more training data. In a scenario ...
13
votes
2answers
5k views

Why are variables of train and test data defined using the capital letter (in Python)?

I hope this question is the most suitable in this site... In Python, usually the class name is defined using the capital letter as its first character, for example ...
0
votes
1answer
1k views

Run Apriori algorithm in python 2.7

I have a DataFrame in python by using pandas which has 3 columns and 80.000.000 rows. The Columns are: {event_id,device_id,category}.[] each device has many events and each event can have more than ...
1
vote
1answer
251 views

Selecting dataset splitting strategy

I found this very informative figure, on how to split the dataset depending on how much data (or how many observations to be more precise) you have. My question is, since "less data" is very ...
1
vote
0answers
129 views

Modeling data combination from multiple sources

I am currently working on my masters thesis and have come to the point where I need to combine data from different sources to find new patterns. An example could be the following: I have transactions ...
2
votes
1answer
347 views

Dataset: language audio clips and country labels

I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from. Does such a dataset exist? Or is there ...
1
vote
3answers
751 views

How to compensate for class imbalance in prediction model?

I'm trying to run a prediction model on a customers' data set to predict the likelihood that a new customer would be interested in buying product X, offered by a company that sells products X,Y and Z. ...
4
votes
1answer
103 views

How to generate bulk graphics using R

I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2. How can I automatically generate one graphic for each ...
1
vote
0answers
138 views

Which Machine Learning Algorithm should be used to predict Waveforms from an already existing collection of waveforms ?

I am new to machine learning and I was wondering what algorithm to use to predict waveforms from an already existing waveform data samples. I have the x coordinates and Y coordinates or (waveforms) ...
1
vote
0answers
364 views

Online noise removal techniques

There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman ...
1
vote
1answer
5k views

Binning data in one of the columns of a dataframe(Using R)

I have a data frame that contains random values between 0-60 (inclusive 0 & 60). These values denote months. I want to bin the data into three categories (x<=6, 6< x <=12, x>12) and ...
1
vote
1answer
68 views

Uploading huge dataset

I have few questions: Is there a website to upload huge research dataset (over 100GB) for free? Which type of compression (rar, zip ... etc) is good for jpeg images? In case of dataset of 120GB. what ...
1
vote
1answer
93 views

How to train on extended data set correctly

I have trained my classifier on pictures with a mixture of several classes on each picture, e.g. A-F. The classifier is able to (nearly) correctly segment those classes on the images. Now I got more ...
1
vote
1answer
4k views

Pandas Query Optimization On Multiple Columns

My training dataset is around 5 MB and test dataset is of the same size. I have run some simulations over the whole dataset couple of times. However, in a latter solution, I ran queries on two columns ...
2
votes
1answer
110 views

Generating data that look alike my original data

If I have a set of N data, each individual data has 4 features. I do not know a priori the relations that could exist (or don't exist) between the features. Is it possible to generate new data from ...
0
votes
2answers
52 views

ideas for variable with branching data?

I had no idea how to explain this in the title but anyway... Let's say I have a data points like this: ...
1
vote
2answers
3k views

Looking for a data set that shows hospital patients' illnesses and their vital signs, during their stay

I'm looking for a data set that shows hospital patients' vital signs (body temperature and/or heart rate, etc.) and their illnesses, over time. Is there such a data set? For example: Patient #1, Male,...
1
vote
1answer
113 views

Specifying neural network output layout for object detection

I have a question about if the modeling of the output detection affects the neural nets capability. In my case I want to train a CNN for object recognition and classification. As an output I want to ...
4
votes
2answers
5k views

Training data set for food image recognition [closed]

There are many excellent posts and answers referencing data sources. However I can't seem to find many for food image/photograph recognition. I would have thought, with so many people taking and ...
1
vote
0answers
38 views

Am I supposed to be using Mixed Effects?

I ran a GLS random effects regression on some NBA data in Stata, and I was told that it was wrong because I didn't use mixed effects model. This may every well be the case, but I was quite confused by ...
0
votes
2answers
345 views

How synthetic sampling of data helps classifier to predict better result?

I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem. I have the idea how to create the synthetic example to bring noticeable accuracy on a ...
3
votes
3answers
1k views

Generating Image data sets for training CNNs

I want to build a system that recognises (with a given uncertainty) the make and model of a car from an image. I have decided to use Convolution Neural Networks, specifically the Caffe framework. My ...
1
vote
0answers
19 views

Is there a real multi-dimension timeseies dataset in which data samples have feature based similarities?

I'm looking for real datasets which are multi-dimension time-series and there exist partial similarities between different categories based on some of the features/dimensions. For example, consider ...
1
vote
0answers
332 views

Is there a parallel to record linkage/entity resolution where ML can be applied to the data for schema matching?

In the context of collecting disparate data sets holding similar information, are their examples of algorithms being able to resolve attributes of records being similar (while their values are ...
2
votes
1answer
3k views
1
vote
1answer
28 views

Best way to gather musical data

I am looking for a relatively large amount of jazz (and other) solos to use for data science purposes - probably somewhere up to a 100 transctipts. Although that is not that much and many pdf ...
4
votes
2answers
415 views

Reducing sample size

I have a large dataset (around $ 10^6 $ samples) and an algorithm that will surely choke on that much data. Suppose that I have removed duplicates and near-duplicates. What are the well-known ...
1
vote
1answer
165 views

How to get Real life Social Network Data?

I am trying to build a case study for Insurance fraud detection using SNA. I have already looked around and build my model using Stanford's SNAP and other similar databases available on the internet ...
0
votes
1answer
66 views

Measuring Difference Between Two Sets of Likert Values

I asked users to complete a likert survey (1-5) at the start of an activity, and conclusion of the activity. What is the best manner to show the rate of change/difference between these two result sets?...
1
vote
1answer
751 views

Handling time series data with gaps

I am working on a dataset with physical measurements taken daily (weight, bmi, etc...) and I am working through the process to graphically represent it. I think it is worth noting that every day has a ...
0
votes
1answer
584 views

Fields list of YearPredictionMSD (identify year of music from Million Song Dataset)

The YearPredictionMSD dataset, is a machine learning practice dataset extracted from the Million Song Dataset and found in the UCI Machine Learning Repository. I couldn't find the field list that ...
1
vote
1answer
105 views

How to test People similarity measure?

I am doing a project on finding famous people who are similar to each other. For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom. ...
2
votes
1answer
233 views

The best way to calculate variations between 2 datasets?

I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database,...
1
vote
4answers
51k views

Remove part of string in R

I have a table in R. It just has two columns and many rows. Each element is a string that contains some characters and some numbers. I need number part of the element. How can I have number part? For ...
0
votes
1answer
365 views

Why is this Binning by Median code wrong?

I was working on Binning by Mean, Median and Boundary in R. ...
3
votes
1answer
298 views

How to sample a statistically uniform dataset

I have a data set of with a distribution that looks something like this: I need to take random sample data from the set so that distribution will be more even. Something like this (take the data in ...
0
votes
1answer
524 views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...