Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
-2
votes
2answers
1k views

Predictive Model with sort-of-non-binary labels

I'm working with government infrastructure public tenders data and want to build a predictive model. I'm want to train a model to predict if a company will or will not participate in a public tender. ...
-1
votes
1answer
167 views

Synonym handling replacement issue with UDF in Apache Spark

We are currently working to replace the synonyms of certain words in a Row. We are facing the problem as shown below. ...
2
votes
4answers
168 views

I have n dimensional data and I want to check integrity, can I downgrade to 2 dimensional feature space via PCA and do so?

Say I have n dimensional data samples. I want to check the integrity of the features, if they are good representation of the respective classes, i.e. these features are good or not. My plan is: I ...
3
votes
2answers
711 views

Predicting hardware failures with limited data

I am exploring using machine learning to predict if a particular hardware component would fail within a timeframe, say 3 months. The ultimate goal is to minimize physical human inspection so that ...
1
vote
3answers
814 views

How to compensate for class imbalance in prediction model?

I'm trying to run a prediction model on a customers' data set to predict the likelihood that a new customer would be interested in buying product X, offered by a company that sells products X,Y and Z. ...
3
votes
2answers
70 views

I want to add demographic data to a data set. Any suggestions on where to find zip code level data?

Ideally, I would like to add demographic features to my data. I have zip codes for each observation, so I was hoping to add in the demographic data by zip code. Unfortunately, I cannot find any ...
3
votes
0answers
352 views

Where can I find a >1K dataset of annotated store receipt pictures/scans?

I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice: vendor date of purchase total amount (...
0
votes
1answer
79 views

City pollution data for data mining project [closed]

I want to do my MSc Project on Data Mining. And I want to work on pollution in cities. Are there accessible and useable pollution data (any type of pollution. e.g. air, water or forest pollution) in ...
2
votes
1answer
428 views

Dataset: language audio clips and country labels

I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from. Does such a dataset exist? Or is there ...
2
votes
1answer
5k views

Pandas Query Optimization On Multiple Columns

My training dataset is around 5 MB and test dataset is of the same size. I have run some simulations over the whole dataset couple of times. However, in a latter solution, I ran queries on two columns ...
1
vote
1answer
960 views

How do I represent a hidden markov model in data structure?

My task involves a POS Tagging using HMM. I am given a training data set (word/tag). I have to write a file with transition probabilities and emission probabilities. I am currently using a nested ...
4
votes
1answer
26k views

Having trouble installing and loading tidyverse- No DIB package [closed]

after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse. I get the following message, seemingly because the package DBI is missing: ...
1
vote
1answer
555 views

GANs to augment training data

I have been reading about Generative Adversarial Networks (GANs) and was wondering if it would make sense to train a generator function only to use it for creating more training data. In a scenario ...
15
votes
2answers
7k views

Why are variables of train and test data defined using the capital letter (in Python)?

I hope this question is the most suitable in this site... In Python, usually the class name is defined using the capital letter as its first character, for example ...
0
votes
1answer
1k views

Run Apriori algorithm in python 2.7

I have a DataFrame in python by using pandas which has 3 columns and 80.000.000 rows. The Columns are: {event_id,device_id,category}.[] each device has many events and each event can have more than ...
1
vote
1answer
298 views

Selecting dataset splitting strategy

I found this very informative figure, on how to split the dataset depending on how much data (or how many observations to be more precise) you have. My question is, since "less data" is very ...
1
vote
1answer
272 views

Sequence of numbers as single feature

Is it possible to use a sequence of numbers as one feature? For example, using libsvm data format: ...
4
votes
1answer
106 views

How to generate bulk graphics using R

I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2. How can I automatically generate one graphic for each ...
0
votes
0answers
168 views

Which Machine Learning Algorithm should be used to predict Waveforms from an already existing collection of waveforms ?

I am new to machine learning and I was wondering what algorithm to use to predict waveforms from an already existing waveform data samples. I have the x coordinates and Y coordinates or (waveforms) ...
1
vote
0answers
367 views

Online noise removal techniques

There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman ...
1
vote
1answer
78 views

Uploading huge dataset

I have few questions: Is there a website to upload huge research dataset (over 100GB) for free? Which type of compression (rar, zip ... etc) is good for jpeg images? In case of dataset of 120GB. what ...
1
vote
1answer
6k views

Binning data in one of the columns of a dataframe(Using R)

I have a data frame that contains random values between 0-60 (inclusive 0 & 60). These values denote months. I want to bin the data into three categories (x<=6, 6< x <=12, x>12) and ...
2
votes
1answer
120 views

Generating data that look alike my original data

If I have a set of N data, each individual data has 4 features. I do not know a priori the relations that could exist (or don't exist) between the features. Is it possible to generate new data from ...
0
votes
2answers
56 views

ideas for variable with branching data?

I had no idea how to explain this in the title but anyway... Let's say I have a data points like this: ...
3
votes
3answers
1k views

Generating Image data sets for training CNNs

I want to build a system that recognises (with a given uncertainty) the make and model of a car from an image. I have decided to use Convolution Neural Networks, specifically the Caffe framework. My ...
3
votes
2answers
6k views

Training data set for food image recognition [closed]

There are many excellent posts and answers referencing data sources. However I can't seem to find many for food image/photograph recognition. I would have thought, with so many people taking and ...
0
votes
2answers
349 views

How synthetic sampling of data helps classifier to predict better result?

I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem. I have the idea how to create the synthetic example to bring noticeable accuracy on a ...
2
votes
3answers
1k views

Multiclass classification with large number of classes but for each user the set of target classes is known

Hi Data Science Stack Exchange! I'm new here but I'm familiar with some machine learning theory (took some courses in school) and my question is more about how to apply ML in a practical setting. I ...
1
vote
0answers
38 views

Am I supposed to be using Mixed Effects?

I ran a GLS random effects regression on some NBA data in Stata, and I was told that it was wrong because I didn't use mixed effects model. This may every well be the case, but I was quite confused by ...
1
vote
0answers
19 views

Is there a real multi-dimension timeseies dataset in which data samples have feature based similarities?

I'm looking for real datasets which are multi-dimension time-series and there exist partial similarities between different categories based on some of the features/dimensions. For example, consider ...
1
vote
3answers
1k views

Where can I get a comprehensive criminal dataset?

I want to create a machine learning model to predict the probability of Person committing crime in future. Given his/her past crime history, age, gender, race, employment history, family information (...
1
vote
0answers
344 views

Is there a parallel to record linkage/entity resolution where ML can be applied to the data for schema matching?

In the context of collecting disparate data sets holding similar information, are their examples of algorithms being able to resolve attributes of records being similar (while their values are ...
2
votes
1answer
594 views

The best way to calculate variations between 2 datasets?

I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database,...
5
votes
2answers
268 views
1
vote
1answer
30 views

Best way to gather musical data

I am looking for a relatively large amount of jazz (and other) solos to use for data science purposes - probably somewhere up to a 100 transctipts. Although that is not that much and many pdf ...
4
votes
2answers
551 views

Reducing sample size

I have a large dataset (around $ 10^6 $ samples) and an algorithm that will surely choke on that much data. Suppose that I have removed duplicates and near-duplicates. What are the well-known ...
1
vote
1answer
174 views

How to get Real life Social Network Data?

I am trying to build a case study for Insurance fraud detection using SNA. I have already looked around and build my model using Stanford's SNAP and other similar databases available on the internet ...
0
votes
1answer
68 views

Measuring Difference Between Two Sets of Likert Values

I asked users to complete a likert survey (1-5) at the start of an activity, and conclusion of the activity. What is the best manner to show the rate of change/difference between these two result sets?...
0
votes
1answer
732 views

Fields list of YearPredictionMSD (identify year of music from Million Song Dataset)

The YearPredictionMSD dataset, is a machine learning practice dataset extracted from the Million Song Dataset and found in the UCI Machine Learning Repository. I couldn't find the field list that ...
1
vote
1answer
816 views

Handling time series data with gaps

I am working on a dataset with physical measurements taken daily (weight, bmi, etc...) and I am working through the process to graphically represent it. I think it is worth noting that every day has a ...
3
votes
1answer
5k views

What is the distribution of categories in imagenet training set (ILSVRC2012)

http://arxiv.org/pdf/1409.0575v3.pdf Table 2 says there are 1,281,167 images and 732-1300 per class in the ILSVRC2012 training set. Ideally I'd like to avoid downloading the 138 GB just for this ...
1
vote
1answer
181 views

How to test People similarity measure?

I am doing a project on finding famous people who are similar to each other. For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom. ...
0
votes
1answer
422 views

Why is this Binning by Median code wrong?

I was working on Binning by Mean, Median and Boundary in R. ...
1
vote
2answers
209 views

How do I know if my training data set is sufficient?

If the number of classes is $C = 5$, training set contains $N = 3000$ samples, and $D = 4$ dimensions. Is my data set is sufficient?
1
vote
1answer
965 views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...
-1
votes
2answers
47k views

Replacing values in multiple columns of a data frame in R

for example consider this dataframe : ...
1
vote
1answer
2k views

Database/warehouse for storing 128D feature vectors

I use openface to extract feature vectors in 128D. I need to find a suitable database engine to store these vectors for future comparisons (calculating the euclidean distance between newly extracted ...
2
votes
2answers
7k views

Classified Enron Email Dataset

I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand ...
6
votes
2answers
3k views

How to handle Memory issues in training Word Embeddings on Large Datasets?

I want to train a word predictability task to generate word embeddings. The document collection contains 243k documents. The code implementation is in torch. I am struggling with the huge size of the ...