Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
5
votes
2answers
3k views

What is the meaning of spherical dataset?

In the following article, one of the statement is as follows: The K-means algorithm is effective only for spherical datasets What does spherical dataset mean?
0
votes
2answers
69 views

How to reduce an unknown size data into a fixed size data? Please read details

Example: Given n number of images marked 1 to n where n is unknown, I can calculate a property of every image which is a scalar quantity. Now I have to represent this property of all images in a fixed ...
1
vote
1answer
44 views

How to create more dimensions for data mining?

I want to analyze product sales data with WEKA. My data have attributes: sell quantity and sales price. I want to add more attributes, such as mean and standard deviation. What formula I should use ...
1
vote
1answer
507 views

In Python, why subsetting with or without square bracket is different? [closed]

Suppose I have a data frame called quoteDF ...
-1
votes
1answer
77 views

How do I coalesce data from various sources?

How do I setup an environment where I can coalesce data from various sources, like mysql databases and csv files, on my macbook? Is there some opensource tool like informatica? or... ?
1
vote
1answer
808 views

What does “Find pattern” mean in data science? [closed]

I am doing certification and i have a project to complete. In project, they have said "Find Patterns". What does it mean? what steps should i carry out?
11
votes
1answer
3k views

Can HDF5 be reliably written to and read from simultaneously by separate python processes?

I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using ...
1
vote
1answer
173 views

how to compare two groups

I need to compare two groups of people, where the independent variable is having a college degree, and the dependent variable is the income. The problem is that if I divide the whole population of ...
1
vote
1answer
46 views

Resources for datasets

Is there a go to source for searching for data sources? Like a way to ask google, specifically to show data files? Specifically, I'm looking for data on the number of smart phones and tablets sold in ...
0
votes
1answer
141 views

Diminishing returns in language identification data set size?

Most problems have a curve whereby the results improve as data are added but level off at some point. Are there research papers or industry results that discuss the correlation between data set size ...
-1
votes
1answer
263 views

How to visualise very large graphs with 250M nodes and 500M+ edges? [closed]

I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files. I am having major difficulties visualising the ...
4
votes
1answer
7k views

How to train an image dataset in TensorFlow? [closed]

As I am new to TensorFlow, I would like to do image recognition in TensorFlow using Python. For this Image Recognition I would like to train my own image dataset and test that dataset. Please answer ...
2
votes
0answers
59 views

Machine Learning over distributed operating system [closed]

I want to implement Distributed operating system mutual exclusion using Machine learning algorithms but I am not getting dataset anywhere to perform. Is there anyway i can get the required dataset? I ...
-1
votes
2answers
71 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
-1
votes
1answer
265 views

ML project ideas for dataset [closed]

Not sure if this is the right forum, but currently i have a dataset which contains a list of TV shows. Each record contains pricing between competitors (price in provider 1. Example: Itunes) TV show ...
1
vote
1answer
1k views

Dataset - Sample pdfs for text processing?

I'm looking for a rather large amount of pdf files for testing my text processing program. Tried looking for an open site to get like some thousand pdfs, but wasn't able to find anything. I don't ...
-1
votes
1answer
5k views

Animal Detection dataset

Update 1: I know that I can seperate animal,bird object annotations/images from imagenet. In fact I already filtered annotations/images that are my interested and downloaded them by URL. I searched ...
1
vote
1answer
400 views

Removing sensitive information from a dataset while retaining its usefulness for machine learning tasks?

I have a dataset that would be useful for training future machine learning models. However, it contains sensitive information (personal identifiers like names and addresses) that make it difficult ...
1
vote
1answer
49 views

System to provide guide to students about getting admissions to universities of their choice or some specific courses [closed]

I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear ...
9
votes
5answers
4k views

Covariate shift detection

Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which ...
3
votes
4answers
578 views

Discrepancy between training set and real-world data set: domain adaptation?

I have read in literature that in some cases the training set is not representative for a real-world dataset. However, I cannot seem to find a proper term describing this phenomenon; what is the ...
1
vote
3answers
6k views

importing csv data in python

I have a csv file with around 130 columns and 6000 rows what is the best way to import them into python, so that I can later use them in a classification algorithm(columns are the labels and rows are ...
1
vote
1answer
810 views

Trouble with Scrapy and Python: ever present unexpected indent

I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. ...
1
vote
0answers
86 views

Data brokers for market-related data, how to choose, where to look for?

Situation I am doing a data science project for my client. Due to NDA I can't disclose anything to the nature of the project, but let's imagine that they manufacture and sell slow cookers. I faced ...
4
votes
3answers
267 views

Data available from industry operations

I'm going to start my degree thesis and I want to do a fault detector system using machine learning techniques. I need datasets for my thesis but I don't know where I can get that data. I'm looking ...
1
vote
1answer
36 views

Techniques to clean the topics

I have set of topics as follows. "web based", "web-based" -> with surplus symbols "technology","technologies" -> with singular and plural "learned", "learnt", "learning" ->suffix stripping Can you ...
1
vote
0answers
27 views

E-Learning related terminology

I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?
5
votes
1answer
169 views

Linear regression on probabilistic data

I have a dataset which is probabilistic in nature. For instance, if a record has d features, then values of $x_1$, ..., $x_d$ are such that their sum adds to 1. In other words, for each $x_i$ data ...
2
votes
1answer
3k views

is there any way to plot ROC curves from weka

i am using some algorithms from weka . i was willing to plot some algorithms' roc curve for comparison . Is it possible and how ?
0
votes
1answer
144 views

Thought process behind data analysis

I am looking for books/tutorials that help you gain the insight into the thought process behind data analysis. Most of the books I've read were mostly documentation - the author shows you a function ...
2
votes
2answers
76 views

How to use a dataset where attribute names are changed?

I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed. My goal is to use dataset ...
0
votes
2answers
65 views

A question on data distribution probability

Let $D={D_k}_1^N$ denotes a set of i.i.d observations, and x denotes random variables, p() denotes probability, whether $p(D|x)$ equals to $p(D)$? Thank you for any help. EDIT: To be more clear, I ...
1
vote
1answer
306 views

Yelp Dataset Archives?

The Yelp Dataset Challenge (https://www.yelp.com/dataset_challenge) releases data for a handful of cities each year. I'd like to analyze some cities from past years. Is this data archived anywhere?
3
votes
0answers
388 views

global flight network (with number of pessengers per year) dataset

Is there such a global flight network dataset that gives you: flight route (the connection between each airports or cities by some flight company) the number of passengers (per year) taking each ...
1
vote
2answers
248 views

Methods to analyze time series data when only small amount of it is labeled

I have time series data out of which only 10% is labeled into 10 classes. What should be the methods or models that I should look for to analyze this problem? I know the question is a bit vague but I ...
0
votes
1answer
2k views

Machine learning for Point Clouds Lidar data

Our main use case is object detection in 3d lidar point clouds i.e. data is not in RGB-D format. We are planning to use CNN for this purpose using theano. Hardware limitations are CPU: 32 GB RAM Intel ...
2
votes
2answers
146 views

How do you ascertain the quality of your data? [closed]

How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such ...
2
votes
2answers
333 views

Data structure design for supporting arbitrary number of columns in table or database

I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data. The task I'm currently struggling is: user needs to ...
0
votes
1answer
69 views

How should I expose data from my app for data scientists?

I'm the product manager for an online app. I'm currently researching a new feature where our users will be able to access "all their raw application data". This data is likely to be used by data ...
4
votes
2answers
4k views

Number of features vs. number of samples : if small sample size is sufficient, why take large number of samples?

As a newbie, I am a little confused. I have a dataset for binary classification with 11 features and 102 sample data. I have seen in most places (e.g., kaggle competions), the dataset may have ...
2
votes
2answers
3k views

Dealing with outliers and z-scores

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques. I am building a function to calculate a social ...
0
votes
2answers
169 views

What tools do you use to clean corrupt data? [closed]

Customer often send currupt data for analysis. I spent a lot of time in cleaning the data or waiting for a correct dataset. Can you recommend a tool that can handle the most common curruptions (eg. ...
-2
votes
2answers
1k views

Predictive Model with sort-of-non-binary labels

I'm working with government infrastructure public tenders data and want to build a predictive model. I'm want to train a model to predict if a company will or will not participate in a public tender. ...
-1
votes
1answer
167 views

Synonym handling replacement issue with UDF in Apache Spark

We are currently working to replace the synonyms of certain words in a Row. We are facing the problem as shown below. ...
2
votes
4answers
168 views

I have n dimensional data and I want to check integrity, can I downgrade to 2 dimensional feature space via PCA and do so?

Say I have n dimensional data samples. I want to check the integrity of the features, if they are good representation of the respective classes, i.e. these features are good or not. My plan is: I ...
3
votes
2answers
710 views

Predicting hardware failures with limited data

I am exploring using machine learning to predict if a particular hardware component would fail within a timeframe, say 3 months. The ultimate goal is to minimize physical human inspection so that ...
1
vote
3answers
814 views

How to compensate for class imbalance in prediction model?

I'm trying to run a prediction model on a customers' data set to predict the likelihood that a new customer would be interested in buying product X, offered by a company that sells products X,Y and Z. ...
3
votes
2answers
70 views

I want to add demographic data to a data set. Any suggestions on where to find zip code level data?

Ideally, I would like to add demographic features to my data. I have zip codes for each observation, so I was hoping to add in the demographic data by zip code. Unfortunately, I cannot find any ...
3
votes
0answers
352 views

Where can I find a >1K dataset of annotated store receipt pictures/scans?

I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice: vendor date of purchase total amount (...
0
votes
1answer
79 views

City pollution data for data mining project [closed]

I want to do my MSc Project on Data Mining. And I want to work on pollution in cities. Are there accessible and useable pollution data (any type of pollution. e.g. air, water or forest pollution) in ...

1
18 19
20
21 22
24