Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
4
votes
2answers
6k views

Training data set for food image recognition [closed]

There are many excellent posts and answers referencing data sources. However I can't seem to find many for food image/photograph recognition. I would have thought, with so many people taking and ...
1
vote
0answers
38 views

Am I supposed to be using Mixed Effects?

I ran a GLS random effects regression on some NBA data in Stata, and I was told that it was wrong because I didn't use mixed effects model. This may every well be the case, but I was quite confused by ...
0
votes
2answers
346 views

How synthetic sampling of data helps classifier to predict better result?

I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem. I have the idea how to create the synthetic example to bring noticeable accuracy on a ...
3
votes
3answers
1k views

Generating Image data sets for training CNNs

I want to build a system that recognises (with a given uncertainty) the make and model of a car from an image. I have decided to use Convolution Neural Networks, specifically the Caffe framework. My ...
1
vote
0answers
19 views

Is there a real multi-dimension timeseies dataset in which data samples have feature based similarities?

I'm looking for real datasets which are multi-dimension time-series and there exist partial similarities between different categories based on some of the features/dimensions. For example, consider ...
1
vote
0answers
340 views

Is there a parallel to record linkage/entity resolution where ML can be applied to the data for schema matching?

In the context of collecting disparate data sets holding similar information, are their examples of algorithms being able to resolve attributes of records being similar (while their values are ...
2
votes
1answer
4k views
1
vote
1answer
28 views

Best way to gather musical data

I am looking for a relatively large amount of jazz (and other) solos to use for data science purposes - probably somewhere up to a 100 transctipts. Although that is not that much and many pdf ...
4
votes
2answers
486 views

Reducing sample size

I have a large dataset (around $ 10^6 $ samples) and an algorithm that will surely choke on that much data. Suppose that I have removed duplicates and near-duplicates. What are the well-known ...
1
vote
1answer
169 views

How to get Real life Social Network Data?

I am trying to build a case study for Insurance fraud detection using SNA. I have already looked around and build my model using Stanford's SNAP and other similar databases available on the internet ...
0
votes
1answer
68 views

Measuring Difference Between Two Sets of Likert Values

I asked users to complete a likert survey (1-5) at the start of an activity, and conclusion of the activity. What is the best manner to show the rate of change/difference between these two result sets?...
1
vote
1answer
784 views

Handling time series data with gaps

I am working on a dataset with physical measurements taken daily (weight, bmi, etc...) and I am working through the process to graphically represent it. I think it is worth noting that every day has a ...
0
votes
1answer
647 views

Fields list of YearPredictionMSD (identify year of music from Million Song Dataset)

The YearPredictionMSD dataset, is a machine learning practice dataset extracted from the Million Song Dataset and found in the UCI Machine Learning Repository. I couldn't find the field list that ...
1
vote
1answer
118 views

How to test People similarity measure?

I am doing a project on finding famous people who are similar to each other. For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom. ...
2
votes
1answer
389 views

The best way to calculate variations between 2 datasets?

I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database,...
1
vote
5answers
71k views

Remove part of string in R

I have a table in R. It just has two columns and many rows. Each element is a string that contains some characters and some numbers. I need number part of the element. How can I have number part? For ...
0
votes
1answer
399 views

Why is this Binning by Median code wrong?

I was working on Binning by Mean, Median and Boundary in R. ...
3
votes
1answer
425 views

How to sample a statistically uniform dataset

I have a data set of with a distribution that looks something like this: I need to take random sample data from the set so that distribution will be more even. Something like this (take the data in ...
1
vote
1answer
703 views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...
1
vote
2answers
176 views

How do I know if my training data set is sufficient?

If the number of classes is $C = 5$, training set contains $N = 3000$ samples, and $D = 4$ dimensions. Is my data set is sufficient?
3
votes
1answer
892 views

How is the evaluation setup for YouTube faces of FaceNet?

The YouTube Faces database (YTF) consists of 3,425 videos of 1,595 different people. Given two videos, the task for YTF is to decide if they contain the same person or not. Having $n$ comparisons, the ...
-1
votes
2answers
39k views

Replacing values in multiple columns of a data frame in R

for example consider this dataframe : ...
1
vote
1answer
2k views

Database/warehouse for storing 128D feature vectors

I use openface to extract feature vectors in 128D. I need to find a suitable database engine to store these vectors for future comparisons (calculating the euclidean distance between newly extracted ...
0
votes
1answer
139 views

Diminishing returns in language identification data set size?

Most problems have a curve whereby the results improve as data are added but level off at some point. Are there research papers or industry results that discuss the correlation between data set size ...
6
votes
1answer
6k views

Plotting different values in pandas histogram with different colors

I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0,1,2). In order to check the distribution of values in each column, I used <...
2
votes
2answers
6k views

Classified Enron Email Dataset

I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand ...
3
votes
0answers
31 views

Merging 2 datawarehouses

Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data ...
1
vote
1answer
48 views

Help reducing a set of features

I am trying to do some clustering. I have a dataset that is very sparse - with the majority of features only occurring in a single vector. Here is a list of our features: https://gist.github.com/...
2
votes
1answer
2k views

Data set size versus data dimension, is there a rule of thumb?

I am trying to collect some data for ML, specifically for training a neural network model, and I don't know how big the data set is enough. So is there a rule of thumb on how many data of dimension <...
0
votes
1answer
117 views

How do I write a SQL (or dwSQL) query across these two different datasets?

I'm having a little fun and I'd love to know how to write a SQL or dwSQL query across these two datasets of Trumps tweets and Facebook posts. Here are the datasets: Trump Facebook Posts https://data....
0
votes
1answer
25 views

How not to see the output in the console when adding a new column in a table?

I have a data table called td and it contains 14k obs and 5k variables. When I run the command td[,fv_mc2:=1] I also get the whole data table as output in my ...
-3
votes
2answers
99 views

Do i have enough ram for the file analysis?

I have a 3GB dataset but my computer only has 4GB ram. Is it possible to run some simple analyses in R-studio (like anova, use some built in packages), etc. with that one 1GB of additional ram? If it ...
3
votes
2answers
291 views

How to transform an imbalanced attribute to make it more suitable for linear regression?

I'm new to data science but trying to get better Here I have an attribute and plotting its histogram From what I know so far such a distribution is imbalanced and my goal is to equalize things a ...
1
vote
0answers
12 views

Which algorthm uses DATASAMPLER for getting X% of the data?

http://docs.orange.biolab.si/3/visual-programming/widgets/data/datasampler.html I would like to know which algorthm uses DATASAMPLER for getting X% of the data? Is it linear congruential method, or ...
2
votes
4answers
4k views

Handling a feature containing multiple values

I have a dataset in following format: Movie ID | Actors | Director | language | ReleaseYear | Genre 1 | Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran | Vidhu Vinod ...
4
votes
1answer
3k views

How to interpret the loading values of a pca?

Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics: ...
-2
votes
2answers
1k views

Predictive Model with sort-of-non-binary labels

I'm working with government infrastructure public tenders data and want to build a predictive model. I'm want to train a model to predict if a company will or will not participate in a public tender. ...
4
votes
1answer
2k views

Training and cross validation error curves

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve. It is observed that the accuracy of training dataset decreases but ...
2
votes
2answers
2k views

Dealing with outliers and z-scores

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques. I am building a function to calculate a social ...
3
votes
1answer
203 views

Shifted feature distribution across different datasets

I am trying to validate a classifier using two different training and testing datasets. The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. ...
2
votes
0answers
107 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
1
vote
3answers
6k views

importing csv data in python

I have a csv file with around 130 columns and 6000 rows what is the best way to import them into python, so that I can later use them in a classification algorithm(columns are the labels and rows are ...
1
vote
1answer
80 views

Transformation from Datawarehouse into Big Data structure

What are the required stages for transferring data in datawarehouse into big data structure. Are there any tools and methods that support it? How to use the schema for such transformation, how to ...
2
votes
0answers
138 views

How to quantitatively compare two or more complex data sets

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response ...
9
votes
1answer
21k views

How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?

I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification ...
1
vote
1answer
601 views

Trouble with Scrapy and Python: ever present unexpected indent

I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. ...
2
votes
2answers
277 views

Feature Engineering

I have a dataframe with around 37,000 rows and 54 columns. Out of these 54 columns two columns namely 'user_id' and 'mail_id' are provided in avery creepy format as shown below: ...
1
vote
4answers
451 views

Discrepancy between training set and real-world data set: domain adaptation?

I have read in literature that in some cases the training set is not representative for a real-world dataset. However, I cannot seem to find a proper term describing this phenomenon; what is the ...
2
votes
0answers
31 views

User ratings and conditional values

I have a set of user transactions with a service and I suspect that there is a strong correlation between how users rate the service and how likely they are to use the service again. I have chosen to ...
0
votes
3answers
213 views

Need to prepare the data to Link Analysis project?

I've a dataset with following schema: ...

1
16 17
18
19 20
22