Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
1
vote
1answer
69 views

Which programming languages should I use to integrate different analysis software packages?

I am currently writing a package to streamline data analysis for a research lab. There are several different analysis software packages that we use, based out of unix, matlab, and (rarely used) python....
-1
votes
1answer
240 views

Splitting Data in scikit-learn

I know how to split the dataset into train and test sets using train_test_split but is there any way that I can split the dataset into three different sets i.e. "Train set", "Test set", and "...
2
votes
3answers
112 views

How do I explain the relationship between Health and Weight?

I have two columns in my dataset: Health and Weight, both being of numeric type: ...
0
votes
0answers
79 views

Is there an image classification dataset which has objects in varying distances?

I would like to analyze / estimate the influence of the architecture on scale invariance in photos. To clarify: It is possible that a network can classify objects well if they are close, but totally ...
1
vote
1answer
47 views

Modeling 3 variable equation

The data produced from my humidity sensor at different temperature is as follows (forget about purpose of frequency and its source) on re-arrangement (temp in deg C) In this case temperature and ...
3
votes
1answer
903 views

How can I look up classes of ImageNet?

After downloading the imagenet urls (link), I see that it is a single 1.1 GB text file which starts like this: ...
6
votes
2answers
6k views

Where can I get labels for small ImageNet?

I have recently downloaded the small image net dataset: http://image-net.org/small/download.php The archives contain a lot of images, but no other files. How do I know which label the images have?
0
votes
1answer
39 views

How do I factor in features which are IDs?

I am an absolute beginner in data science and I had this (possibly stupid) question on my mind, while reading a problem in Kaggle: Say I'm given IDs of some clients, IDs of products that they sell, ...
0
votes
1answer
721 views

SMOTE and standardisation

I have an unbalanced dataset X. I split it between data and labels, then I standardize the data. Then I use train_test_split to split between train and test data and I output the result. Now I want ...
1
vote
1answer
143 views

Uncertainty calculation through integration and correct analysis methodology

In the following text, uncertainty refers to standard deviation. My methodology and notation was based mainly on the book "A Student's Guide to Data and Error Analysis" by H.J.C. Berendsen." ...
0
votes
2answers
978 views

How to find nearest neighbors in SMOTE

I am reading the original paper by Chawla and others for SMOTE. I am trying to understand how to generate these synthetic examples for over-sampling the minority class. The paper says: "Synthetic ...
0
votes
1answer
81 views

SAS procedure IF THEN ELSE loop did not recognize the Roman letters or numbers as value

I have written the following SAS procedure to display Grades based on the salary: Not Working: This displays only 'I' for all observations irrespective of the else condition that says any salary less ...
3
votes
3answers
763 views

Where can I find datasets with large number of features (100000) but less samples(100-1000)?

I am working on a dimensionality reduction algorithm for datasets with high feature to sample ratio. I need datasets with around 100000 features or more and 100-1000 samples with class labels to ...
1
vote
1answer
102 views

A faster multidimensional scaling?

I've been playing around with some data, and I've created a decently large similarity matrix (each column value for a row represents how similar the document corresponding to the column is to the ...
2
votes
0answers
117 views

Linux tools to handle large data sets (several TB) [closed]

I come from a physics background that has a very specific tool (ROOT, a C++ toolkit) to handle very large data sets for a very specific purpose. For smaller data sets (MB to GB), I use modules in ...
2
votes
0answers
122 views

Good Libraries or Software for Temporal Bayesian Network Structure Learning?

I would like to infer structures of BNs with edges within and between time slices by using data. I tried libpgm (http://pythonhosted.org/libpgm/) and found that 2TBN is ideal for my purpose, but ...
4
votes
0answers
238 views

Are SAS Data Storage Options designed for Big Data?

I understand that SAS has some data storage options in the form of Default Storage tables or SAS datasets - in the SAS library. Multi-dimensional Storage cubes - in the OLAP Server. Third Party ...
0
votes
1answer
74 views

How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets?

I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(...
0
votes
1answer
30 views

Convert Lat /Lon of User input to Lat/Lon of Open Data

I have data from a public data set in gridded form 2.5 degree x 2.5 degree(lat,lon). Latitude goes from 90 N to -90 S. Longitude goes from 0 to 357.5. It is stored every 2.5 degrees and there are no ...
3
votes
2answers
8k views

problem loading data into R

I have problem with loading data into R: ...
1
vote
1answer
72 views

Characterisitcs of the data set for a binary classification problem

I want to build a classifier for my problem statement and for that I don't have data. So while doing data acquisition, what should be the minimum sample size? And would it be a good practice if I ...
1
vote
0answers
194 views

Generating a text training dataset from a grammar

I want to generate documents based on a grammar to build a custom training database. What are the tools and techniques to generate random texts based on a given grammar. More specifically, I would ...
1
vote
1answer
1k views

Why does measuring for Asymmetric Binary Attributes exclude t (Values which are 0 for both)?

I'm teaching myself Data mining and now struggling from this problem. According to this table(http://slidewiki.org/slide/22352), if the attributes are Asymmetric, dissimilarity is calculated without ...
4
votes
2answers
3k views

How to handle Memory issues in training Word Embeddings on Large Datasets?

I want to train a word predictability task to generate word embeddings. The document collection contains 243k documents. The code implementation is in torch. I am struggling with the huge size of the ...
2
votes
2answers
197 views

Technical name for this data wrangling process? Multiple columns into multi-factor single column

What is the technical name for the following data wrangling process? I want to collapse Table A into Table B. (To make the data suitable for ANOVA.) Table A: ...
1
vote
3answers
72k views

How to group by multiple columns in dataframe using R and do aggregate function

I have a dataframe with columns as defined below. I have provided one set of example, similar to this I have many countries with loan amount and gender variables ...
0
votes
1answer
5k views

How to read Several JSON files to a dataframe in R?

I have a folder with 30,000 plus JSON file. A sample file with contents is posted below. ...
0
votes
1answer
167 views

Verification of trained system

I have trained a system in order to detect some features from a set of scenarios. Now the system can detect and classify that set. How can I validate how that system works in real world? What ...
10
votes
3answers
12k views

When should we consider a dataset as imbalanced?

I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced. My question is, are there any rules of thumb that tell us when we should subsample the large ...
3
votes
1answer
3k views

What is the distribution of categories in imagenet training set (ILSVRC2012)

http://arxiv.org/pdf/1409.0575v3.pdf Table 2 says there are 1,281,167 images and 732-1300 per class in the ILSVRC2012 training set. Ideally I'd like to avoid downloading the 138 GB just for this ...
1
vote
3answers
857 views

Where can I get a comprehensive criminal dataset?

I want to create a machine learning model to predict the probability of Person committing crime in future. Given his/her past crime history, age, gender, race, employment history, family information (...
9
votes
6answers
22k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
1
vote
0answers
64 views

How to model this variable?

I'm doing some data prep on a dataset provided by a telecommunication company. There is a continuous variable that indicates how many months have passed since a customer renewed her contract. However, ...
1
vote
0answers
315 views

Python: How to make model predict in a generalized manner using ML Algorithm

I have a dataset for which I am trying to predict target variables. ...
3
votes
1answer
110 views

How to classify support call texts?

I have a spreadsheet with thousands of records regarding support requests. Case number, Issue description, etc. Our goal is to classify these records in many categories in order to assign them the ...
2
votes
3answers
1k views

How big should training data be?

I am solving a problem connected with medicine and from each patient I get about 100 features. It's a classification diseases problem, however measurements take a lot of time and also require money. ...
2
votes
2answers
231 views

How to go about obtaining data

I am a researcher working on traffic flow prediction. For my project I would need to obtain historical traffic flow raw data. Google Maps has a traffic flow overlay which is pretty much the kind of ...
9
votes
2answers
4k views

Benchmark datasets for collaborative filtering

I'd like to test a new algorithm for collaborative filtering. A typical use case is to recommend movies based on the preferences of users similar to the specific user. What are some common benchmark ...
0
votes
1answer
51 views

Add new factor across multiple groups

I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for ...
7
votes
1answer
6k views

How do you calculate how dense or sparse a dataset is?

I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf In order to select ...
2
votes
3answers
978 views

Multiclass classification with large number of classes but for each user the set of target classes is known

Hi Data Science Stack Exchange! I'm new here but I'm familiar with some machine learning theory (took some courses in school) and my question is more about how to apply ML in a practical setting. I ...
1
vote
0answers
74 views

What's the best way to rank aggregate imdb rating data?

I have an average rating of all votes as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?: 1) multiply each average rating by the total ...
2
votes
0answers
49 views

How to interpret a dataset with a pseudo-logarithmic trend?

I am dealing with a scatterplot where I am trying to figure out the relationship between two variables, but I have so far failed to identify what the best fit could be. The presence of zero values ...
19
votes
3answers
12k views

is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of gather(), separate() and ...
0
votes
1answer
49 views

How can I access dataset from Nasa website

So Now I was working on my project ("Failure prediction models in FoF") and the Tool I will be using is RapidMinder but I need to have a dataset to run the tool on, I found a nice description of ...
3
votes
1answer
534 views

Determining completeness of dataset

I'm hoping you have some research or experience with determining the completeness of a data set. I'm trying to use a twitter dataset I scraped myself and want to have an indication on the completeness....
1
vote
2answers
72 views

Gathering the number of Google results from a large amount of searches.

I am trying to build a simple dataset using Google, mainly because it seems like the best option for what I want. I want to measure fame for a large group of scientists. The quick method is to ...
2
votes
0answers
126 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
-1
votes
2answers
312 views

Finding databases with same columns and different content/ model

I am looking for datasets that have the same columns and their content (rows) are different. When I run a decision tree (classification) I'll get different models (trees) for each of them. It can be ...
2
votes
3answers
2k views

Sentiment Analysis: Creating dictionary from dataset

I've collected around half a million of unmarked comments from a newspage. The newspage has anti-foreign background. Because of that, a relativley high number of comments contain hate. Do you have ...