Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
2
votes
1answer
101 views

Working with text files in Excel

I have an excel file containing a long text in column A. I am looking for the words starting by "popul" such as popular and populate . I can find these cells by the formula: ...
2
votes
3answers
1k views

Reading Persian Characters in R

I have a csv file with characters in Persian and I cannot view them in R correctly. Also, I cannot subset based on Persian characters values. Here is a sample code: ...
0
votes
1answer
2k views

How to generate ratings without training data?

I am working on generating restaurant ratings automatically and I have various feature values like delivery time, cost estimate, etc. I want to generate a rating for each restaurant between 0 to 5. ...
2
votes
0answers
478 views

Where can I download a tagged dataset of text related to finance, programming, analytics etc.? [closed]

I want to create and train a model which classifies a new text content into finance, programming, analytics, design etc. Where can I get a relevant dataset to train my models? TIA.
2
votes
3answers
160 views

What possible data products can be built using this dataset [closed]

I have a dataset which contains information about when do people enter and leave a premise. I have the following information in the dataset : Person Id Time of Entry Time of Leaving The dataset has ...
2
votes
2answers
126 views

How do you ascertain the quality of your data? [closed]

How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such ...
3
votes
1answer
113 views

Looking for smallest set of rows that form a natural key in a data set

I have several sets of text files on hdfs that are exports from relations. Unfortunately I do not know the structure of the table is, but I do know that each has a multi-part key that defines a row ...
12
votes
2answers
1k views

Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?

I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing. As I would say my knowledge about more complex analysis is quite limited I've done ...
2
votes
1answer
108 views

Data cleaning: Relationships between columns

I have a training data set distributed in two files. File 1: This contains actual classification for each X1. X1 is unique in this file. X1 has one-to-one relationship with X2, i.e. X2 is also unique....
0
votes
1answer
84 views

new and fresh ideas with stack exchange dataset [closed]

I'm a new bee for data analysis . i need to work on a research project in big data analysis. first of all i search for a dataset and i found interest in stack exchange data dump. however i browse for ...
4
votes
1answer
888 views

What types of features are used in a large-scale click-through rate prediction problem?

Something that I often see in papers (example) about large-scale learning is that click-through rate (CTR) problems can have up to a billion of features for each example. In this Google paper the ...
2
votes
1answer
3k views

What is the best way to scale a numerical dataset [closed]

I have a dataset with differents attributes which don't have the same range on their values which is a problem when we need to compute distance beetween objects. After some research i found that i can ...
12
votes
4answers
11k views

Downloading a large dataset on the web directly into AWS S3

Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL? Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
1
vote
1answer
671 views

LFR as synthesized data for dynamic community detection methods

In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper ...
19
votes
3answers
13k views

How to generate synthetic dataset using machine learning model learnt with original dataset?

Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset ...
0
votes
1answer
410 views

Identity covariance matrix, decorrelated data?

Why would you want to decorrelated data? As I am reading about PCA and whitening on image data for DNN, I wonder what is the purpose of achieving the identity covariance matrix in your data is? Is ...
1
vote
1answer
1k views

Time series change rate calculations for displaying trend line chart

I'm struggling to find a solution to produce a line chart which shows a trend from a Time Series metric. I try to make my line chart look similar to the following: Basically, I want to make use of ...
1
vote
1answer
667 views

Measuring similarity for sets with same cardinality

The Jaccard coefficient measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets. I had 100 sets all of same ...
2
votes
2answers
145 views

How to make an effective sampling from a database of text documents?

Problem: I want to know methods to perform an effective sampling from a database. The size of the database is about 250K text documents and in this case each text ...
1
vote
3answers
721 views

Transposing Every nth row to column in a large dataset

I am attempting to work with a very large data-set (~1.5mil lines) for the first time in SAS and I am having some difficulty. The data-set I have is formatted as a "long" .txt file as follows: ...
1
vote
2answers
2k views

Split columns by finding “£” and then converting to minvalue, maxvalue

I have a pandas dataframe with a salary column which contains values like: £36,000 - £40,000 per year plus excellent bene..., £26,658 to £32,547 etc I isolated this column and split it with the ...
1
vote
1answer
680 views

training neural net with multiple sets of time-series data

I have the following data ($x^1_i$, $y^1_i$) for $i=1,2,...N_1$ ($x^2_i$, $y^2_i$) for $i=1,2,...N_2$ ... ($x^m_i$, $y^m_i$) for $i=1,2,...N_m$ Is it possible to train a neural net to produce ...
2
votes
1answer
1k views

InterquartileRange takes up most instances in data set

I'm very new to this community, so please overlook my noobness. I have a data set with 2948 instances and I tried to remove outliers using InterquartileRange filter in Weka. The issue is that the ...
2
votes
3answers
896 views

OCR / Text Recognition and Recovery Problem

I am working on a research project that deals with American military casualties during WWII. Specifically, I am attempting to construct a count of casualties for each service at the county level. ...
0
votes
1answer
2k views

Machine learning for Point Clouds Lidar data

Our main use case is object detection in 3d lidar point clouds i.e. data is not in RGB-D format. We are planning to use CNN for this purpose using theano. Hardware limitations are CPU: 32 GB RAM Intel ...
4
votes
3answers
257 views

Data available from industry operations

I'm going to start my degree thesis and I want to do a fault detector system using machine learning techniques. I need datasets for my thesis but I don't know where I can get that data. I'm looking ...
3
votes
1answer
953 views

Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?

I am currently working on a multi-class classification problem with a large training set. However, it has some specific characteristics, which induced me to experiment with it, resulting in few ...
4
votes
2answers
192 views

Where did this NY Times op-ed get his Google Search data?

I hope this is a question appropriate for SO. The article in question: http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html As far as I can tell, the only ...
1
vote
2answers
673 views

Appropriate way to store data in R

I have data, which looks like this: These data are only for one subject. I will have a lot more. These data will be analyzed in R. Now I'm storing them like this: ...
0
votes
2answers
267 views

Data transposition code in R

I've been working in SAS for a few years but as my time as a student with a no-cost-to-me license comes to an end, I want to learn R. Is it possible to transpose a data set so that all the ...
8
votes
3answers
941 views

Interactive Graphing while logging data

I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
1
vote
0answers
82 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
4
votes
2answers
543 views

API for Company Data Enrichment Suggestions

I'm looking for API suggestions for enriching data on companies. Currently I use the Crunchbase API to look up a company's name or domain and I am trying to gather the domain/name (if I don't already ...
6
votes
1answer
190 views

Evaluating Recommendation engines

What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare ...
2
votes
1answer
252 views

Confused about description of YearPrediction Dataset

https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD According to the description given in the above link, the Attribute information specifies "average and covariance over all 'segments', each ...
1
vote
2answers
231 views

DBPedia as Table not having all the properties

I browsed a sample for available data at http://dbpedia.org/page/Sachin_Tendulkar. I wanted these properties as columns, so I downloaded the CSV files from http://wiki.dbpedia.org/DBpediaAsTables. ...
2
votes
0answers
299 views

Creating Data model for mahout recommendation engine

I am trying to build an item-item similarity matching recommendation engine with mahout. The data set is as in the following format ( attributes are in text not in numerals format ) ...
3
votes
2answers
886 views

API for historical housing prices

I'm looking for an (ideally free) API that would have time series avg/median housing prices by zip code or city/state. Quandl almost fits the bill, but it returns inconsistent results across ...
4
votes
2answers
130 views

Reduction of multiple answers to single variable

The questionnaire for the data is here The first question takes multiple entry for the same question, I want to reduce this to a single variable. How do I do it? The clean data is available here. NB:...
2
votes
3answers
8k views

Large categorical dataset for regression

I need to collect several large datasets (thousands of samples, dozens of features) for regression with only categorical inputs. I already look for such datasets in the UCI repository, but I did not ...
6
votes
2answers
788 views

Matrix properties and machine learning/data mining

I'm doing some data analysis in a Statistical Pattern Recognition course using PRML. We analyzed a lot of matrix properties, like eigenvalues, column independence, positive semi-definite matrix, etc. ...
3
votes
1answer
771 views

MovieLens data set

I want to analyze MovieLens data set and load on my machine the M1 file. I combine actually two data files (ratings.dat and movies.dat) and sort the table according ...
4
votes
3answers
2k views

Single Layer Perceptron with three classes

I need some help with a single layered perceptron with multiple classes. What I need to do is classify a dataset with three different classes, by now I just learnt how to do it with two classes, so I ...
5
votes
2answers
120 views

Amalgamating multiple datasets with different variables coding

I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables ...
3
votes
3answers
10k views

How to classify and cluster this time series data

I have post already the question few months ago about my project that I'm starting to work on. This post can be see here: Human activity recognition using smartphone data set problem Now, I know ...
1
vote
1answer
61 views

Versatile data structure for combined statistics

Not sure if this is Math, Stats or Data Science, but I figured I would post it here to get the site used. As a programmer, when you have a system/component implemented, you might want to allow some ...
3
votes
1answer
102 views

Trouble representing a problem

I have a problem and I'm having trouble representing it - first I thought I should use graph theory (nodes and edges) and now I'm not sure. My data is some tanks names and it's volumes, those tanks ...
29
votes
4answers
12k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
1
vote
0answers
30 views

Freebase Related Models

It may be unlikely that anyone knows this but I have a specific question about Freebase. Here is the Freebase page from the Ford Taurus automotive model . It has a property called "Related Models". ...
3
votes
1answer
3k views

Data sets for evaluating text retrieval quality [closed]

I'm searching for data sets for evaluating text retrieval quality. TF-IDF is a popular similarity measure, but is it the best choice? And which variant is the best choice? Lucenes Scoring for example ...