Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dataset]

A dataset is a collection of data, generally represented in tabular form, with columns signifying different variables and rows signify different members of the set.

0
votes
1answer
21 views

How to save dataframe in csv format without data corrupt?

Since the data size is large, I tried to save the dataframe after preprocessing so that I can load it for later use. I saved my dataframe train_df of length 293000 ...
0
votes
0answers
10 views

How do I extract data out of cornell movie dialog corpus?

I am new in making chatbots.I want to make a chatbot for which I need a dataset.I decided to go with cornell movie dialog corpus. But I am unable to understand the structure of their data . You can ...
0
votes
0answers
4 views

Looking for a dataset on (online) newspapers, news aggregators, and other kinds of news outlets [migrated]

I intend to calculate a virality-score of current news. For this purpose I will crawl major newspapers, news aggregators and other kinds of news outlets. Does anyone here know about a dataset ...
0
votes
2answers
19 views

Question on creating your own dataset [on hold]

I am new to the machine learning world, i want to create my own dataset for computer vision body measurement. I have this program that can create 3d bodies based on several body measurements such as ...
0
votes
1answer
8 views

Change values from nominal to numeric

I want to change the values of the class labels from nominal into numeric. e.g if the values of a class are {iris-setosa,iris-virginica,iris-versicolor} i want to make them {0,1,2} so the instances ...
1
vote
1answer
15 views

Understanding a dataset (prior to applying ML models) with no metadata given

How do you understand a dataset when there is no metadata given (no details about the attributes given in the dataset)? It is difficult to comprehend the attribute names as only the short forms are ...
0
votes
1answer
24 views

Is there any way how to make samples balanced?

I have a dataset which consists of attributes on breakdown of machines.The target variable is machine status which are populated with ones and zeros. The distribution of ones and zeros are given below ...
0
votes
1answer
11 views

Attribute selection from dataset

I work on a dataset with numeric values.The class labels has also numeric values.I made the 6 numeric class labels into one witch contain values like e.g happy_pleased. I want to insert the new .arff ...
0
votes
1answer
37 views

What could be a dataset in which the presence of an outlier dramatically affects the performance of Ordinary Least Squares (OLS) regression?

I am tasked with giving an example of a dataset in which the presence of an outlier dramatically affects the performance of Ordinary Least Squares (OLS) regression. I've searched and searched the web ...
0
votes
0answers
16 views

how to save a time-series dataset

i am looking for a way to store a dataset of time series data efficiently with python. For my last project i have used numpy and npz files for saving image data. At this project there are about 15000 ...
0
votes
2answers
12 views

Meaning of stratify parameter

I'm training a Neural Network and I'm trying to divide my data into training and testing sets. I have a lot of output classes and for some of them I have as little as 2 examples, so I would like to ...
1
vote
0answers
16 views

How do I prepare data in which each output row depends on multiple input rows?

My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y. In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on ...
0
votes
1answer
29 views

How to use two different datasets as train and test sets?

Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. The problem that I've encountered, now that I'm trying to make my own ...
0
votes
0answers
14 views

How to prepare data for time series RNN

I want to use RNN on my data. The data is from a number of medical devices, it is a time series. My problem is that there are few types of files, each having data from a deferent source: File #1: ...
0
votes
1answer
20 views

Is it correct to use non-target values of test set to engineer new features for train set?

Suppose, I have a dataset with a feature_1 value and a target value. Now, I want to engineer a new feature by creating relative value by subtracting mean from each value. Question: Can I (1) use ...
0
votes
1answer
21 views

LSTM - How to prepare train from a dataset which contains multiple observations for different events

I m using LSTM in a project related to MobiFall dataset which contains falls and daily activitives - such as walking, sitting etc - sensed by accelerometer, gyroscope and orientation sensors in x,y,z ...
1
vote
0answers
29 views

Semantic segmentation dataset inspection

Is there any tool that can calculate the statistics of semantic segmentation dataset (and possibly the similar types of datasets ...)? So that it shows the number of classes, the total number of ...
0
votes
0answers
10 views

Using character level model to notes generation

I would like to use character based model for music generation. Predict note after N notes. (like character model based text generators). Do to this I would like to use RNN with LSTM in Keras. I ...
0
votes
1answer
27 views

Skewed two class data set

Is there any theory on the influence of skew in the data set on the performance of binary classifiers? At work, we are doing abuse detection, the negative population is regular logins, and the ...
0
votes
0answers
19 views

arXiv API: parsing the full paper and load data in ElasticSearch

I want to make some analysis with ElasticSearch and Kibana using the arXiv API. As far as I've seen the arXiv API allows to extract almost everything about an article, like the date, authors, ...
0
votes
1answer
23 views

how to split available data into training and testing (Information security)

I was advised to ask my question here. Recently, I made a post about finding suitable dataset for SIEM (Security Information and Event Management) systems. The goal was to work on classification and ...
0
votes
1answer
13 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
0
votes
0answers
14 views

Dataset for ML project: Application of Machine Learning to improve breast cancer treatment

My issue in relation to finding the right dataset for my current project. I visited the cancer imaging website to find some data for investigating my hypothesis [which is " denser breast tissue (fibro-...
1
vote
2answers
33 views

Round numbers in dataset [closed]

How do I round all numbers in the following dataset to two decimals? Data: (made with sklearn.datasets.make_classification) ...
1
vote
0answers
28 views

Dataset Creation for Images

I am creating an image dataset of objects. We have 15 classes of objects and need to provide a color to the objects also. What ...
1
vote
0answers
16 views

How to prepare photo data for training model to recognize bowling ball name, brand and manufacturer from photo of bowling ball?

I am asked to do this. The client can only prepare a only one photo for each ball from the product page of bowling manufacturer. However, I need huge amount of image data for each ball. Here is ...
0
votes
1answer
27 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
0answers
17 views

Handle outliers, Losing many data by removing natural outliers

I have 2 skewed features, here is the summary of one of the features ...
0
votes
2answers
27 views

Delete rows of a column using another column statistics in R [closed]

I have a dataframe with the following columns: If a year (see "year" column) has less than 10 weeks reported, I would like to delete it from the dataframe. The column "weeks_reporting" gives the ...
1
vote
1answer
26 views

Training dataset decreasing in quality (Google data science blog)

I have a complex algorithm that decides when it should show customers of an only shop an ad on our website, after they log in, in hope that they will buy what is in the ad. We have no control what is ...
1
vote
1answer
19 views

Difference between bagging and boosting

Can anyone explain me the basic difference between bagging and boosting and which technique can be used in which scenario?
0
votes
3answers
53 views

How to access substrings in pandas column and store it into new columns?

I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like ...
1
vote
0answers
16 views

Splitting a geospatial data set with existing groupings

Data set: I have a data set consisting of trip data for a large number of cars. The raw data consists of rows with (GPS) coordinates, timestamp, current fuel consumption, temperature, etc. It has been ...
0
votes
1answer
9 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretized. However in a computer it's impossible to have values $\in \mathbb{R}$, so what this mean ? Thanks
1
vote
2answers
39 views

Feature engineering decrease my cross validation

I'm currently working on a fraud detection data set. I'm evaluating my training data with a 10-skfold roc auc and an estimator of default param LightGBM. But, the problem is every time I try to create ...
0
votes
1answer
24 views

Prediction interval for very small dataset

Does it make sense to calculate prediction interval for very small dataset (about 60 samples)? I know it's easy for linear regression. But it seems like for tree-based methods or non-linear methods it ...
2
votes
1answer
93 views

How to correctly perform data sampling for train/test split in multi-label dataset?

Problem statement I have a text multi-label classification dataset, and I've found a problem with the dataset sampling. I'm facing two different strategies. The first one consists in preprocessing ...
0
votes
0answers
7 views

Does integrating data points that fall under a curve make sense?

I have data I've collected from the fanfiction website, Archive of Our Own. I want to know if there's a relationship between the length of a chapter and the average number of hits per chapter. I ...
1
vote
0answers
31 views
0
votes
0answers
18 views

Folder Structure — Where to store data

Most of the python cookie-cutters that I've seen for data science (I've been looking at cookiecutter-data-science) and reading elsewhere, everything seems to indicate that its best practice to keep ...
1
vote
0answers
18 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
0
votes
0answers
25 views

Working with many unique values in a column

I have only learnt basics of data science about 4 months ago. And now I am working on project. This is my code: https://pastebin.com/er1EA7pZ This is my data: https://www.kaggle.com/grosvenpaul/...
1
vote
0answers
13 views

Buy key word audio data set

I am trying to implement audio recognition that can sport key words. I came across google speech dataset but it has a limited data set. Is it possible to buy a new keyword data set ? Can you guys ...
2
votes
1answer
27 views

Creating a single number from a numpy array - Python [closed]

I am working on a gender classification project. I am extracting the pixels of an image using a Numpy array in Python, similar to the one below: ...
0
votes
0answers
23 views

Multiple grouping in R [on hold]

I would like to group the parts of pipeline by the name (OZNAKA_ODS) and lenghth(Dolzina) of the pipeline. Further i would like to write a new atribute table with only identical names of pipelineand ...
1
vote
0answers
20 views

How to measure retention statistics?

I have a dataset with the ID, name, joining date, leaving date as features. I was asked to measure employee retention and health of it. What can I derive from these? What are some latest trends and ...
2
votes
2answers
184 views

Create a binary-classification dataset (python: sklearn.datasets.make_classification)

I would like to create a dataset, however I need a little help. The dataset is completely fictional - everything is something I just made up. Since the dataset is for a school project, it should be ...
2
votes
0answers
18 views

detecting anomaly from cdr data

I need to find anomalies from this. The term anomaly in my case is in which square id and in which time the total activity goes higher, how can I detect anomalies with the help of machine learning ...
2
votes
2answers
26 views

How to get a count the number of observations for each year with a Pandas datetime column?

I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year. This is the column: ...