Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Population Data

I'm trying to creating a heatmap using Esri Arcgis but need to find a data set of the population within a 1-mile and 3-mile radius. Does anyone know if there is a free data set that I could use? ...
0
votes
1answer
19 views

How to set and check normal distribution on a data set?

Sorry if my question is simple, I have a data set with two class and want to check and set normal distribution on it(if it was necessary, in MATLAB). But the question is that I should use it on every ...
1
vote
0answers
15 views

Bayesian network in Python: both construction and sampling

For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some ...
0
votes
1answer
28 views

How much is the Class Imbalance Problem rates?

I'm working on a data set and wanted to know is there a standard rate about Class Imbalance problem or not? I have 47 samples in Class A and 150 Sample in class B , should I use Class Imbalance ...
0
votes
0answers
14 views

How to best use Large images in training set for deep learning

I would like to ask you about how I should deal with the images I have. They are really large. They have this shape: (3000, 4000, 3). I'm working on a multilabel classification model. And I want to ...
1
vote
0answers
50 views

Load in many files from google drive into colab

I am trying to load in 30k images (600mb) from Google drive into Google Colaboratory to further process them with Keras/PyTorch. Therefore I have first mounted my Google drive using: ...
1
vote
1answer
96 views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
0
votes
0answers
6 views

Is there a dataset that contains images of talking people with information about their mouth state ( open, closed, teeth showing, etc.)?

Is there a dataset that contains images of talking people with information about their mouth state ( open, closed, teeth showing, etc.)? I would like to classify images based on whether or not a ...
0
votes
1answer
36 views

How to scale a variable when not knowing the maximum

I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target). However, normalizing between 0-1 for instance means ...
-1
votes
2answers
66 views

What is the approx minimum size of dataset required to build 90% correct model?

I am working with a financial dataset size which is around 3000. I have attempted the supervised-learning regression techniques and not able to go beyond 70% accuracy. Features: 10 Data size:3700 ...
0
votes
1answer
8 views

Face dataset organized by folder [closed]

I'm looking for a quite little/medium dataset (from 50MB to 500MB) that contains photos of famous people organized by folder. The tree structure have to bee something like this: ...
1
vote
0answers
42 views

How to deal with large datasets?

I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even ...
0
votes
0answers
10 views

Wavelet transform to identify 'meaningful' data changes

Can I use the wavelet transform to find the 'meaningful' changes in a series of 1d data? For example I have a series of integers representing angles of line segments. The majority of these values are ...
0
votes
1answer
39 views

Which Machine Learning methods would be good for a beginner-level Data Science project for described dataset? [closed]

I want to do my first beginner-level project regarding Data Science. I picked my data set which is described below. I want to make a Python script which will make some predictions on our data set ...
0
votes
2answers
59 views

How to handle addresses of the restaurants to feed the data-set in the ML model?

I have data from different restaurants which have also address of the restaurants now I want to predict the food delivery timing based on the given data, now the restaurant address is one of the ...
0
votes
0answers
24 views

Audio dataset preprocessing to perform cry detection

I am building a neural network to perform cry detection (i.e., binary classification of cry/non-cry situations) when capturing sound in a house environment. To do so, I performed the following steps: ...
2
votes
1answer
51 views

Could I add a one hot encoding to each feature representing “has data” versus “has no data”

I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and ...
0
votes
1answer
32 views

The distribution of dataset train and test are the differents, how to fix this?

I am new in data science and like some help to understand my problem. For instance, I have two signals non-stationary for the same condition (figure 1). I acquisition them at different times(in the ...
0
votes
0answers
25 views

Anonymizing the CSV data using Python Faker

To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations. Is it possible to anonymize the ...
0
votes
1answer
28 views

Stata: Trying to merge two data sets with common year observations

I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
0
votes
1answer
15 views

How much of a disadvantage is a small sample size?

I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my ...
0
votes
0answers
25 views

How to make a heatmap using keypoints from MPII data set

So I am tiring to figure this out but keep running into a wall. So what I want to do is create a heatmap from a 2D image based on the key points generated from the MPII dataset. I have found a CSV ...
0
votes
0answers
12 views

Space and time fourier transform of data

I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
1
vote
0answers
16 views

Finding frequency as a function of wavevector ( Fourier Space)

I have data for a function $G(x,y,t)$ where $x,y$ show space position (2D) and $t$ is time. I know from Fourier transformation definition that: $G(x,y,t)=\int q \:dq \: d\omega \: d\theta\: \hat{G}(q,...
0
votes
0answers
13 views

What units are depth prediction dataset such as kitti or nyu in? km? meters?

What units are depth prediction dataset such as kitti or nyu in? km? meters? In general, when running a depth estimation network, do we pass in ground truth as km or m or some other more normalized ...
1
vote
0answers
20 views

Column of Lists into Regex vs. Tuple

I'm working on preparing/analysing this dataset of Russian propaganda ads, in which there are columns which contain lists. 'interests_categories', for example, has a row which contains ['Unknown', '...
0
votes
0answers
15 views

Free historical news data base?

I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
1
vote
3answers
30 views

How to extract insights from the given data?

Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in ...
0
votes
0answers
18 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
3
votes
2answers
234 views

Need machine learning algorithm to fill in time-series data

I am currently dealing with a time-series data set with cyclical gaps every 30 minutes (30 minutes of data, 30 minutes of no data). Is there a relatively simple way of using scikit-learn (or some ...
0
votes
1answer
180 views

Sampling labeled data for anomaly detection

I'm currently working on a project that requires the use of unsupervised anomaly detection, but I'm unable to find a relavent data set, so I'm considering the following option: Assuming I have a data ...
2
votes
1answer
34 views

Dataset from sequence of messages

I have a sorted dataset by datestamp which looks like this: ...
1
vote
1answer
23 views

How to represent a dataset as a linked list-like graph?

I would like to visualize this data set using Python and probably a visualization tool like Matplotlib. The data set contains three columns: a user id with a question, a user id with an answer, and ...
0
votes
2answers
31 views

What's the meaning of precomputed features?

When i learn about deep learning, I found dataset with precomputed features form. Link (http://cs.stanford.edu/people/karpathy/deepimagesent/coco.zip). What's the different with usual dataset?
1
vote
3answers
215 views

How to anonymize (de-identify) data in Python?

I have tried a simple algorithm to anonymize the data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values of strings ...
0
votes
0answers
13 views

How can I use MIT CBCL dataset in python

I have downloaded the MIT CBCL face dataset, but I can't find any code online on how to use it in python. the MIT data set is in two formats .greynorm and .pgm format which I don't know how to use any ...
0
votes
0answers
11 views

How to write custom de-identification algorithm in Python?

I have tried a simple algorithm to anonymize my data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values. The data ...
0
votes
1answer
220 views

How to save API data into CSV format?

Is there an easy way to convert API data with python commands and output the results in CSV format? Here's what I'm doing now: ...
1
vote
1answer
32 views

Why do train, test, validation datasets need to have the same distribution?

I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution. Can ...
1
vote
1answer
165 views

MemoryError when concatenating a large data-frame

I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns. My issue is that these CSV files are quite large and when ...
0
votes
0answers
20 views

How to avoid dataset shift or covariate shift? [duplicate]

I wonder If there can occur a dataset shift or covariate shift through the following train_test_split-method? ...
1
vote
1answer
25 views

What dataset was Stanford NER trained on?

I would like to re-train the Stanford NER library from scratch as a 1 class model. Only 3,4 and 7 class models are available out of the box. Is it possible to obtain the data that the model was ...
0
votes
0answers
37 views

What's the size of Google's complete Conceptual Captions image captioning dataset with all the images downloaded from the listed URLs?

The original dataset provided by Google, here, consists of 'Image URL - Caption' pairs in both the provided training and validation sets. I have to work on an image captioning project and wanted to ...
0
votes
0answers
6 views

Dataset for Title Detection and Hierarchization

I am working on a project which deals with detecting Headings in a PDF and developing a hierarchy out of documents(similar to Table Of Contents Generation).I have read some related papers(1,2).But ...
1
vote
0answers
17 views

How can I measure data anonymity when present summary statistics

I am looking to present users with aggregate statistics of a highly sensitive underlying dataset. See example data below: ...
0
votes
1answer
20 views

Recommendations to generate a training set of images

We are about to start a series of experiments in physics which will generate 8.5 thousands of images per month. This is a study on microscopic material reactions so the images are not 'normal' objects ...
1
vote
1answer
144 views

can I use z-score normalization even if it doesn't make sense for my data to be negative?

I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
1answer
27 views

Is it possible to identify what data a neural net was trained on? [closed]

For example, if there are 5 major data sets for a certain topic. Would it be possible to see if a deployed network was trained off of one them? How would one approach this?
0
votes
1answer
19 views

Dataset : Handling rows where some column value is not applicable

I am working on some LTE Data say of shape (100000, 10) . Now suppose I have a column Jitter Avg. Now Jitter occurs only for any connected call and for call which could not be setup due to some ...
1
vote
1answer
79 views

K-fold-cross-validation if training dataset is much smaller than test dataset?

I'm a beginner in machine learning and I have a special case in which I have only a small training dataset of about 500 images and a test dataset of 10,000 images. Does it still make sense to do a 10-...