Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
1
vote
1answer
105 views

How to test People similarity measure?

I am doing a project on finding famous people who are similar to each other. For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom. ...
2
votes
1answer
234 views

The best way to calculate variations between 2 datasets?

I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database,...
1
vote
4answers
51k views

Remove part of string in R

I have a table in R. It just has two columns and many rows. Each element is a string that contains some characters and some numbers. I need number part of the element. How can I have number part? For ...
0
votes
1answer
366 views

Why is this Binning by Median code wrong?

I was working on Binning by Mean, Median and Boundary in R. ...
3
votes
1answer
302 views

How to sample a statistically uniform dataset

I have a data set of with a distribution that looks something like this: I need to take random sample data from the set so that distribution will be more even. Something like this (take the data in ...
0
votes
1answer
534 views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...
1
vote
2answers
157 views

How do I know if my training data set is sufficient?

If the number of classes is $C = 5$, training set contains $N = 3000$ samples, and $D = 4$ dimensions. Is my data set is sufficient?
3
votes
1answer
876 views

How is the evaluation setup for YouTube faces of FaceNet?

The YouTube Faces database (YTF) consists of 3,425 videos of 1,595 different people. Given two videos, the task for YTF is to decide if they contain the same person or not. Having $n$ comparisons, the ...
-1
votes
2answers
31k views

Replacing values in multiple columns of a data frame in R

for example consider this dataframe : ...
1
vote
1answer
1k views

Database/warehouse for storing 128D feature vectors

I use openface to extract feature vectors in 128D. I need to find a suitable database engine to store these vectors for future comparisons (calculating the euclidean distance between newly extracted ...
0
votes
1answer
138 views

Diminishing returns in language identification data set size?

Most problems have a curve whereby the results improve as data are added but level off at some point. Are there research papers or industry results that discuss the correlation between data set size ...
5
votes
1answer
5k views

Plotting different values in pandas histogram with different colors

I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0,1,2). In order to check the distribution of values in each column, I used <...
2
votes
2answers
6k views

Classified Enron Email Dataset

I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand ...
3
votes
0answers
30 views

Merging 2 datawarehouses

Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data ...
1
vote
1answer
48 views

Help reducing a set of features

I am trying to do some clustering. I have a dataset that is very sparse - with the majority of features only occurring in a single vector. Here is a list of our features: https://gist.github.com/...
2
votes
1answer
2k views

Data set size versus data dimension, is there a rule of thumb?

I am trying to collect some data for ML, specifically for training a neural network model, and I don't know how big the data set is enough. So is there a rule of thumb on how many data of dimension <...
0
votes
1answer
111 views

How do I write a SQL (or dwSQL) query across these two different datasets?

I'm having a little fun and I'd love to know how to write a SQL or dwSQL query across these two datasets of Trumps tweets and Facebook posts. Here are the datasets: Trump Facebook Posts https://data....
0
votes
1answer
25 views

How not to see the output in the console when adding a new column in a table?

I have a data table called td and it contains 14k obs and 5k variables. When I run the command td[,fv_mc2:=1] I also get the whole data table as output in my ...
-3
votes
2answers
96 views

Do i have enough ram for the file analysis?

I have a 3GB dataset but my computer only has 4GB ram. Is it possible to run some simple analyses in R-studio (like anova, use some built in packages), etc. with that one 1GB of additional ram? If it ...
3
votes
2answers
248 views

How to transform an imbalanced attribute to make it more suitable for linear regression?

I'm new to data science but trying to get better Here I have an attribute and plotting its histogram From what I know so far such a distribution is imbalanced and my goal is to equalize things a ...
1
vote
0answers
12 views

Which algorthm uses DATASAMPLER for getting X% of the data?

http://docs.orange.biolab.si/3/visual-programming/widgets/data/datasampler.html I would like to know which algorthm uses DATASAMPLER for getting X% of the data? Is it linear congruential method, or ...
2
votes
4answers
4k views

Handling a feature containing multiple values

I have a dataset in following format: Movie ID | Actors | Director | language | ReleaseYear | Genre 1 | Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran | Vidhu Vinod ...
4
votes
1answer
3k views

How to interpret the loading values of a pca?

Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics: ...
-1
votes
2answers
1k views

Predictive Model with sort-of-non-binary labels

I'm working with government infrastructure public tenders data and want to build a predictive model. I'm want to train a model to predict if a company will or will not participate in a public tender. ...
4
votes
1answer
2k views

Training and cross validation error curves

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve. It is observed that the accuracy of training dataset decreases but ...
2
votes
2answers
2k views

Dealing with outliers and z-scores

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques. I am building a function to calculate a social ...
3
votes
1answer
169 views

Shifted feature distribution across different datasets

I am trying to validate a classifier using two different training and testing datasets. The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. ...
2
votes
0answers
106 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
1
vote
3answers
6k views

importing csv data in python

I have a csv file with around 130 columns and 6000 rows what is the best way to import them into python, so that I can later use them in a classification algorithm(columns are the labels and rows are ...
1
vote
1answer
78 views

Transformation from Datawarehouse into Big Data structure

What are the required stages for transferring data in datawarehouse into big data structure. Are there any tools and methods that support it? How to use the schema for such transformation, how to ...
2
votes
0answers
129 views

How to quantitatively compare two or more complex data sets

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response ...
7
votes
1answer
20k views

How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?

I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification ...
1
vote
1answer
432 views

Trouble with Scrapy and Python: ever present unexpected indent

I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. ...
2
votes
2answers
269 views

Feature Engineering

I have a dataframe with around 37,000 rows and 54 columns. Out of these 54 columns two columns namely 'user_id' and 'mail_id' are provided in avery creepy format as shown below: ...
1
vote
4answers
371 views

Discrepancy between training set and real-world data set: domain adaptation?

I have read in literature that in some cases the training set is not representative for a real-world dataset. However, I cannot seem to find a proper term describing this phenomenon; what is the ...
2
votes
0answers
29 views

User ratings and conditional values

I have a set of user transactions with a service and I suspect that there is a strong correlation between how users rate the service and how likely they are to use the service again. I have chosen to ...
0
votes
3answers
198 views

Need to prepare the data to Link Analysis project?

I've a dataset with following schema: ...
1
vote
1answer
69 views

Which programming languages should I use to integrate different analysis software packages?

I am currently writing a package to streamline data analysis for a research lab. There are several different analysis software packages that we use, based out of unix, matlab, and (rarely used) python....
-1
votes
1answer
239 views

Splitting Data in scikit-learn

I know how to split the dataset into train and test sets using train_test_split but is there any way that I can split the dataset into three different sets i.e. "Train set", "Test set", and "...
2
votes
3answers
112 views

How do I explain the relationship between Health and Weight?

I have two columns in my dataset: Health and Weight, both being of numeric type: ...
0
votes
0answers
78 views

Is there an image classification dataset which has objects in varying distances?

I would like to analyze / estimate the influence of the architecture on scale invariance in photos. To clarify: It is possible that a network can classify objects well if they are close, but totally ...
1
vote
1answer
47 views

Modeling 3 variable equation

The data produced from my humidity sensor at different temperature is as follows (forget about purpose of frequency and its source) on re-arrangement (temp in deg C) In this case temperature and ...
2
votes
1answer
883 views

How can I look up classes of ImageNet?

After downloading the imagenet urls (link), I see that it is a single 1.1 GB text file which starts like this: ...
6
votes
2answers
6k views

Where can I get labels for small ImageNet?

I have recently downloaded the small image net dataset: http://image-net.org/small/download.php The archives contain a lot of images, but no other files. How do I know which label the images have?
0
votes
1answer
39 views

How do I factor in features which are IDs?

I am an absolute beginner in data science and I had this (possibly stupid) question on my mind, while reading a problem in Kaggle: Say I'm given IDs of some clients, IDs of products that they sell, ...
0
votes
1answer
709 views

SMOTE and standardisation

I have an unbalanced dataset X. I split it between data and labels, then I standardize the data. Then I use train_test_split to split between train and test data and I output the result. Now I want ...
1
vote
1answer
139 views

Uncertainty calculation through integration and correct analysis methodology

In the following text, uncertainty refers to standard deviation. My methodology and notation was based mainly on the book "A Student's Guide to Data and Error Analysis" by H.J.C. Berendsen." ...
0
votes
2answers
957 views

How to find nearest neighbors in SMOTE

I am reading the original paper by Chawla and others for SMOTE. I am trying to understand how to generate these synthetic examples for over-sampling the minority class. The paper says: "Synthetic ...
0
votes
1answer
81 views

SAS procedure IF THEN ELSE loop did not recognize the Roman letters or numbers as value

I have written the following SAS procedure to display Grades based on the salary: Not Working: This displays only 'I' for all observations irrespective of the else condition that says any salary less ...
3
votes
3answers
740 views

Where can I find datasets with large number of features (100000) but less samples(100-1000)?

I am working on a dimensionality reduction algorithm for datasets with high feature to sample ratio. I need datasets with around 100000 features or more and 100-1000 samples with class labels to ...