Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
1
vote
1answer
48 views

Help reducing a set of features

I am trying to do some clustering. I have a dataset that is very sparse - with the majority of features only occurring in a single vector. Here is a list of our features: https://gist.github.com/...
3
votes
1answer
3k views

Data set size versus data dimension, is there a rule of thumb?

I am trying to collect some data for ML, specifically for training a neural network model, and I don't know how big the data set is enough. So is there a rule of thumb on how many data of dimension <...
2
votes
2answers
236 views

How to go about obtaining data

I am a researcher working on traffic flow prediction. For my project I would need to obtain historical traffic flow raw data. Google Maps has a traffic flow overlay which is pretty much the kind of ...
0
votes
1answer
119 views

How do I write a SQL (or dwSQL) query across these two different datasets?

I'm having a little fun and I'd love to know how to write a SQL or dwSQL query across these two datasets of Trumps tweets and Facebook posts. Here are the datasets: Trump Facebook Posts https://data....
0
votes
1answer
26 views

How not to see the output in the console when adding a new column in a table?

I have a data table called td and it contains 14k obs and 5k variables. When I run the command td[,fv_mc2:=1] I also get the whole data table as output in my ...
-3
votes
2answers
100 views

Do i have enough ram for the file analysis?

I have a 3GB dataset but my computer only has 4GB ram. Is it possible to run some simple analyses in R-studio (like anova, use some built in packages), etc. with that one 1GB of additional ram? If it ...
4
votes
1answer
4k views

How to interpret the loading values of a pca?

Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics: ...
1
vote
1answer
190 views

Uncertainty calculation through integration and correct analysis methodology

In the following text, uncertainty refers to standard deviation. My methodology and notation was based mainly on the book "A Student's Guide to Data and Error Analysis" by H.J.C. Berendsen." ...
5
votes
1answer
2k views

Training and cross validation error curves

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve. It is observed that the accuracy of training dataset decreases but ...
3
votes
1answer
1k views

How can I look up classes of ImageNet?

After downloading the imagenet urls (link), I see that it is a single 1.1 GB text file which starts like this: ...
1
vote
1answer
82 views

Transformation from Datawarehouse into Big Data structure

What are the required stages for transferring data in datawarehouse into big data structure. Are there any tools and methods that support it? How to use the schema for such transformation, how to ...
0
votes
1answer
86 views

SAS procedure IF THEN ELSE loop did not recognize the Roman letters or numbers as value

I have written the following SAS procedure to display Grades based on the salary: Not Working: This displays only 'I' for all observations irrespective of the else condition that says any salary less ...
9
votes
1answer
23k views

How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?

I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification ...
2
votes
0answers
32 views

User ratings and conditional values

I have a set of user transactions with a service and I suspect that there is a strong correlation between how users rate the service and how likely they are to use the service again. I have chosen to ...
1
vote
1answer
72 views

Which programming languages should I use to integrate different analysis software packages?

I am currently writing a package to streamline data analysis for a research lab. There are several different analysis software packages that we use, based out of unix, matlab, and (rarely used) python....
1
vote
2answers
73 views

Gathering the number of Google results from a large amount of searches.

I am trying to build a simple dataset using Google, mainly because it seems like the best option for what I want. I want to measure fame for a large group of scientists. The quick method is to ...
1
vote
1answer
47 views

Modeling 3 variable equation

The data produced from my humidity sensor at different temperature is as follows (forget about purpose of frequency and its source) on re-arrangement (temp in deg C) In this case temperature and ...
0
votes
1answer
42 views

How do I factor in features which are IDs?

I am an absolute beginner in data science and I had this (possibly stupid) question on my mind, while reading a problem in Kaggle: Say I'm given IDs of some clients, IDs of products that they sell, ...
9
votes
3answers
978 views

Interactive Graphing while logging data

I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
34
votes
4answers
14k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
2
votes
3answers
1k views

How big should training data be?

I am solving a problem connected with medicine and from each patient I get about 100 features. It's a classification diseases problem, however measurements take a lot of time and also require money. ...
1
vote
1answer
105 views

A faster multidimensional scaling?

I've been playing around with some data, and I've created a decently large similarity matrix (each column value for a row represents how similar the document corresponding to the column is to the ...
2
votes
0answers
124 views

Good Libraries or Software for Temporal Bayesian Network Structure Learning?

I would like to infer structures of BNs with edges within and between time slices by using data. I tried libpgm (http://pythonhosted.org/libpgm/) and found that 2TBN is ideal for my purpose, but ...
2
votes
0answers
140 views

Linux tools to handle large data sets (several TB) [closed]

I come from a physics background that has a very specific tool (ROOT, a C++ toolkit) to handle very large data sets for a very specific purpose. For smaller data sets (MB to GB), I use modules in ...
3
votes
0answers
264 views

Are SAS Data Storage Options designed for Big Data?

I understand that SAS has some data storage options in the form of Default Storage tables or SAS datasets - in the SAS library. Multi-dimensional Storage cubes - in the OLAP Server. Third Party ...
9
votes
4answers
3k views

Suggest text classifier training datasets

Which freely available datasets can I use to train a text classifier? We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
0
votes
1answer
77 views

How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets?

I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(...
9
votes
2answers
926 views

How to model user's buying behavior on Amazon?

For our final course project in Data Science, we proposed the following- Give the Amazon Reviews Dataset, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that ...
3
votes
2answers
10k views

problem loading data into R

I have problem with loading data into R: ...
0
votes
1answer
34 views

Convert Lat /Lon of User input to Lat/Lon of Open Data

I have data from a public data set in gridded form 2.5 degree x 2.5 degree(lat,lon). Latitude goes from 90 N to -90 S. Longitude goes from 0 to 357.5. It is stored every 2.5 degrees and there are no ...
1
vote
1answer
77 views

Characterisitcs of the data set for a binary classification problem

I want to build a classifier for my problem statement and for that I don't have data. So while doing data acquisition, what should be the minimum sample size? And would it be a good practice if I ...
1
vote
0answers
217 views

Generating a text training dataset from a grammar

I want to generate documents based on a grammar to build a custom training database. What are the tools and techniques to generate random texts based on a given grammar. More specifically, I would ...
0
votes
1answer
6k views

How to read Several JSON files to a dataframe in R?

I have a folder with 30,000 plus JSON file. A sample file with contents is posted below. ...
0
votes
1answer
171 views

Verification of trained system

I have trained a system in order to detect some features from a set of scenarios. Now the system can detect and classify that set. How can I validate how that system works in real world? What ...
3
votes
1answer
111 views

How to classify support call texts?

I have a spreadsheet with thousands of records regarding support requests. Case number, Issue description, etc. Our goal is to classify these records in many categories in order to assign them the ...
1
vote
0answers
64 views

How to model this variable?

I'm doing some data prep on a dataset provided by a telecommunication company. There is a continuous variable that indicates how many months have passed since a customer renewed her contract. However, ...
1
vote
0answers
321 views

Python: How to make model predict in a generalized manner using ML Algorithm

I have a dataset for which I am trying to predict target variables. ...
0
votes
1answer
57 views

Add new factor across multiple groups

I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for ...
3
votes
1answer
681 views

Determining completeness of dataset

I'm hoping you have some research or experience with determining the completeness of a data set. I'm trying to use a twitter dataset I scraped myself and want to have an indication on the completeness....
10
votes
1answer
8k views

How do you calculate how dense or sparse a dataset is?

I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf In order to select ...
1
vote
0answers
85 views

What's the best way to rank aggregate imdb rating data?

I have an average rating of all votes as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?: 1) multiply each average rating by the total ...
2
votes
0answers
51 views

How to interpret a dataset with a pseudo-logarithmic trend?

I am dealing with a scatterplot where I am trying to figure out the relationship between two variables, but I have so far failed to identify what the best fit could be. The presence of zero values ...
0
votes
1answer
49 views

How can I access dataset from Nasa website

So Now I was working on my project ("Failure prediction models in FoF") and the Tool I will be using is RapidMinder but I need to have a dataset to run the tool on, I found a nice description of ...
-1
votes
2answers
315 views

Finding databases with same columns and different content/ model

I am looking for datasets that have the same columns and their content (rows) are different. When I run a decision tree (classification) I'll get different models (trees) for each of them. It can be ...
7
votes
2answers
1k views

Difference between training and test data distributions

A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate ...
2
votes
0answers
148 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
7
votes
2answers
7k views

For which real world data sets does DBSCAN surpass K-means.?

For clustering, DBSCAN surpass k-means in terms of handling arbitrary shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data ...
4
votes
2answers
434 views

Tools to perform SQL analytics on 350TB of csv data

In short, what would be the best method/tricks/techniques/tools for performing ad hoc sql (style) queries on 350TB of csv data? Would there be other options, tool wise that would do it faster if we ...
8
votes
3answers
7k views

One Hot encoding for large number of values

How do we use one hot encoding if the number of values which a categorical variable can take is large ? In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
2
votes
3answers
175 views

how to extract high frequency phrase

This is the corpus abcdef abcd xabcd b bcef ef aef what I want to extract is abcd, ef, ...