Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,463
questions
0
votes
1
answer
93
views
new and fresh ideas with stack exchange dataset [closed]
I'm a new bee for data analysis . i need to work on a research project in big data analysis. first of all i search for a dataset and i found interest in stack exchange data dump. however i browse for ...
- 109
4
votes
1
answer
923
views
What types of features are used in a large-scale click-through rate prediction problem?
Something that I often see in papers (example) about large-scale learning is that click-through rate (CTR) problems can have up to a billion of features for each example. In this Google paper the ...
- 143
2
votes
1
answer
6k
views
What is the best way to scale a numerical dataset [closed]
I have a dataset with differents attributes which don't have the same range on their values which is a problem when we need to compute distance beetween objects. After some research i found that i can ...
- 410
16
votes
5
answers
24k
views
Downloading a large dataset on the web directly into AWS S3
Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?
Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
- 163
1
vote
1
answer
761
views
LFR as synthesized data for dynamic community detection methods
In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper ...
- 379
22
votes
3
answers
14k
views
How to generate synthetic dataset using machine learning model learnt with original dataset?
Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset ...
- 323
0
votes
1
answer
529
views
Identity covariance matrix, decorrelated data?
Why would you want to decorrelated data?
As I am reading about PCA and whitening on image data for DNN, I wonder what is the purpose of achieving the identity covariance matrix in your data is?
Is ...
1
vote
1
answer
2k
views
Time series change rate calculations for displaying trend line chart
I'm struggling to find a solution to produce a line chart which shows a trend from a Time Series metric.
I try to make my line chart look similar to the following:
Basically, I want to make use of ...
- 113
1
vote
1
answer
1k
views
Measuring similarity for sets with same cardinality
The Jaccard coefficient measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets.
I had 100 sets all of same ...
- 113
2
votes
2
answers
243
views
How to make an effective sampling from a database of text documents?
Problem: I want to know methods to perform an effective sampling from a database.
The size of the database is about 250K text documents and in this case each text ...
- 21
1
vote
3
answers
2k
views
Transposing Every nth row to column in a large dataset
I am attempting to work with a very large data-set (~1.5mil lines) for the first time in SAS and I am having some difficulty. The data-set I have is formatted as a "long" .txt file as follows:
...
- 11
1
vote
2
answers
2k
views
Split columns by finding "£" and then converting to minvalue, maxvalue
I have a pandas dataframe with a salary column which contains values like:
£36,000 - £40,000 per year plus excellent bene...,
£26,658 to £32,547 etc
I isolated this column and split it with the ...
- 11
1
vote
2
answers
953
views
training neural net with multiple sets of time-series data
I have the following data
($x^1_i$, $y^1_i$) for $i=1,2,...N_1$
($x^2_i$, $y^2_i$) for $i=1,2,...N_2$
...
($x^m_i$, $y^m_i$) for $i=1,2,...N_m$
Is it possible to train a neural net to produce ...
- 13
2
votes
1
answer
1k
views
InterquartileRange takes up most instances in data set
I'm very new to this community, so please overlook my noobness.
I have a data set with 2948 instances and I tried to remove outliers using InterquartileRange filter in Weka. The issue is that the ...
- 123
2
votes
3
answers
1k
views
OCR / Text Recognition and Recovery Problem
I am working on a research project that deals with American military casualties during WWII. Specifically, I am attempting to construct a count of casualties for each service at the county level. ...
- 21
0
votes
1
answer
2k
views
Machine learning for Point Clouds Lidar data
Our main use case is object detection in 3d lidar point clouds i.e. data is not in RGB-D format. We are planning to use CNN for this purpose using theano. Hardware limitations are CPU: 32 GB RAM Intel ...
4
votes
3
answers
278
views
Data available from industry operations
I'm going to start my degree thesis and I want to do a fault detector system using machine learning techniques. I need datasets for my thesis but I don't know where I can get that data. I'm looking ...
- 143
3
votes
1
answer
1k
views
Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?
I am currently working on a multi-class classification problem with a large training set. However, it has some specific characteristics, which induced me to experiment with it, resulting in few ...
- 133
4
votes
2
answers
210
views
Where did this NY Times op-ed get his Google Search data?
I hope this is a question appropriate for SO.
The article in question: http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html
As far as I can tell, the only ...
- 153
1
vote
2
answers
826
views
Appropriate way to store data in R
I have data, which looks like this:
These data are only for one subject. I will have a lot more.
These data will be analyzed in R.
Now I'm storing them like this:
...
- 135
0
votes
2
answers
547
views
Data transposition code in R
I've been working in SAS for a few years but as my time as a student with a no-cost-to-me license comes to an end, I want to learn R.
Is it possible to transpose a data set so that all the ...
9
votes
3
answers
1k
views
Interactive Graphing while logging data
I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
2
votes
0
answers
89
views
Cross-sell models and additional holders
I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products.
Up to now I was only taking into considerations the clients as first holders....
- 81
5
votes
2
answers
630
views
API for Company Data Enrichment Suggestions
I'm looking for API suggestions for enriching data on companies. Currently I use the Crunchbase API to look up a company's name or domain and I am trying to gather the domain/name (if I don't already ...
- 151
8
votes
2
answers
269
views
Evaluating Recommendation engines
What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare ...
- 1,890
3
votes
1
answer
347
views
Confused about description of YearPrediction Dataset
https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD
According to the description given in the above link,
the Attribute information specifies "average and covariance over all 'segments', each ...
- 69
2
votes
2
answers
256
views
DBPedia as Table not having all the properties
I browsed a sample for available data at http://dbpedia.org/page/Sachin_Tendulkar. I wanted these properties as columns, so I downloaded the CSV files from http://wiki.dbpedia.org/DBpediaAsTables.
...
2
votes
0
answers
337
views
Creating Data model for mahout recommendation engine
I am trying to build an item-item similarity matching recommendation engine with mahout. The data set is as in the following format ( attributes are in text not in numerals format )
...
- 1,890
3
votes
2
answers
1k
views
API for historical housing prices
I'm looking for an (ideally free) API that would have time series avg/median housing prices by zip code or city/state.
Quandl almost fits the bill, but it returns inconsistent results across ...
- 131
4
votes
2
answers
137
views
Reduction of multiple answers to single variable
The questionnaire for the data is here
The first question takes multiple entry for the same question, I want to reduce this to a single variable. How do I do it?
The clean data is available here.
NB:...
- 41
4
votes
3
answers
10k
views
Large categorical dataset for regression
I need to collect several large datasets (thousands of samples, dozens of features) for regression with only categorical inputs. I already look for such datasets in the UCI repository, but I did not ...
- 1,737
8
votes
2
answers
981
views
Matrix properties and machine learning/data mining
I'm doing some data analysis in a Statistical Pattern Recognition course using PRML. We analyzed a lot of matrix properties, like eigenvalues, column independence, positive semi-definite matrix, etc. ...
- 81
4
votes
1
answer
811
views
MovieLens data set
I want to analyze MovieLens data set and load on my machine the M1 file. I combine actually two data files (ratings.dat and movies.dat) and sort the table according ...
- 313
4
votes
3
answers
4k
views
Single Layer Perceptron with three classes
I need some help with a single layered perceptron with multiple classes.
What I need to do is classify a dataset with three different classes, by now I just learnt how to do it with two classes, so I ...
5
votes
2
answers
134
views
Amalgamating multiple datasets with different variables coding
I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables ...
- 51
4
votes
2
answers
13k
views
How to classify and cluster this time series data [duplicate]
I have post already the question few months ago about my project that I'm starting to work on. This post can be see here:
Human activity recognition using smartphone data set problem
Now, I know ...
- 401
1
vote
1
answer
73
views
Versatile data structure for combined statistics
Not sure if this is Math, Stats or Data Science, but I figured I would post it here to get the site used.
As a programmer, when you have a system/component implemented, you might want to allow some ...
- 113
3
votes
1
answer
108
views
Trouble representing a problem
I have a problem and I'm having trouble representing it - first I thought I should use graph theory (nodes and edges) and now I'm not sure.
My data is some tanks names and it's volumes, those tanks ...
- 143
35
votes
4
answers
16k
views
Quick guide into training highly imbalanced data sets
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
- 5,434
1
vote
0
answers
38
views
Freebase Related Models [closed]
It may be unlikely that anyone knows this but I have a specific question about Freebase. Here is the Freebase page from the Ford Taurus automotive model . It has a property called "Related Models". ...
- 183
3
votes
1
answer
4k
views
Data sets for evaluating text retrieval quality [closed]
I'm searching for data sets for evaluating text retrieval quality.
TF-IDF is a popular similarity measure, but is it the best choice? And which variant is the best choice? Lucenes Scoring for example ...
- 341
1
vote
0
answers
37
views
Apps to manage/host data sets [closed]
I'm looking for the best solution to manage and host datasets for journalistic pursuits. I am assessing https://www.documentcloud.org and http://datahub.io/.
Can anyone explain the differences ...
- 111
3
votes
1
answer
221
views
Query similarity: how much data is used in practice?
I recently read Similarity Measures for Short Segments of Text (Metzler et al.). It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their ...
- 811
7
votes
5
answers
5k
views
Where can I find free spatio-temporal dataset for download?
Where can I find free spatio-temporal dataset for download so that I can play with it in R ?
- 171
1
vote
0
answers
43
views
Subgraph isomorphism and Anti-monotone property
While finding frequent subgraphs in single large graph, subgraph isomorphism (test) is not considered because its not anti-monotone. How and why subgraph isomorphism is not anti-monotone ?
- 111
4
votes
4
answers
331
views
Handling huge dataset imbalance (2 class values) and appropriate ML algorithm
I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem ...
- 803
15
votes
2
answers
5k
views
Analyzing A/B test results which are not normally distributed, using independent t-test
I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution.
In fact the distribution resembles more closely the Landau Distribution.
I ...
- 253
3
votes
1
answer
6k
views
R aggregate() with dates
I am working on a data set that has multiple traffic speed measurements per day. My data is from the city of chicago, and it is taken every minute for about six months. I wanted to consolidate this ...
- 91
28
votes
3
answers
43k
views
Data Science Project Ideas [closed]
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.
I have just started with Data Science and ...
- 383
59
votes
6
answers
16k
views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
- 693