Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
2
votes
0answers
182 views

Will flattening multivariate time series data before clustering make the results meaningless?

I have a large number of financial time series that I wish to do cluster analysis on. Each time series has the same length and spans multiple years of daily data (returns, volatility, etc.). As part ...
1
vote
0answers
56 views

Can anyone direct me on how to got to [closed]

I am a newbie to python. I have uploaded about 2000 images into my google drive, assigned a variable to all content in the folder and tried to split the folder's content into training and testing, ...
3
votes
1answer
126 views

ASR on low dataset

I am doing an ASR(automatic speech recognition) as master thesis on low key dataset. Voice and text data is labelled. There are around 4000 phrases and around 5 hours speech. I don't have background ...
1
vote
0answers
25 views

Replace values before and after a certain value [closed]

I have a variable like this: V1: 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 I want to change the 0 before the 1 to 1, and the 0 after the 2 to 2. It is supposed to look like this: V1: 1 1 1 1 1 1 1 0 0 0 0 ...
3
votes
1answer
559 views

For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
7
votes
1answer
2k views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
2
votes
1answer
314 views

Rescaling each image Individually with keras

I am a beginner working on a simple CNN to classify X-ray detector images. Due to source intensity, all images have different max values. I want to use ImageDataGenerator to rescale those images to be ...
2
votes
3answers
533 views

How to handle “year” variable for Machine Learning models

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable? Thanks ...
2
votes
1answer
77 views

Format of data in SQL for machine learning

I am a beginner at Machine Learning and am starting out on a ML project. I have a large chunk of the source material and have started extracting the data from it to be stored in SQL (initial test with ...
1
vote
1answer
49 views

One scaler for all features or one scaler per feature?

I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?
0
votes
1answer
97 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
1
vote
2answers
150 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
2
votes
1answer
129 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
1
vote
0answers
14 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
1
vote
3answers
237 views

Correlation between categorical variables based on the target distribution

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
1
vote
0answers
20 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
1
vote
1answer
166 views

Improving the performace of the Naive Bayes classifier by decorrelating the data

I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $...
0
votes
1answer
38 views

How much of a disadvantage is a small sample size?

I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my ...
1
vote
0answers
112 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
1
vote
0answers
22 views

Column of Lists into Regex vs. Tuple

I'm working on preparing/analysing this dataset of Russian propaganda ads, in which there are columns which contain lists. 'interests_categories', for example, has a row which contains ['Unknown', '...
1
vote
2answers
38 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
14
votes
2answers
24k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
2
votes
1answer
326 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
2
votes
0answers
125 views

How to label time series data correctly for training RNN/CNN models?

My Case I want to tackle a deep learning classification task using various smartphone sensor data. I will use a self-built data acquisition app and basically walk around with the phone, manually ...
1
vote
0answers
135 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
0answers
41 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
0
votes
0answers
64 views

GPS Data Preprocessing Recommandations?

GPS Data Preprocessing I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
1answer
13 views

Should I have “normal” sampled data in my dataset?

I am busy working on a project to find the reasons why kids in normal households are doing badly in school. I have a dataset of which consists of kids that live in environments where the family is ...
3
votes
2answers
4k views

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
2
votes
2answers
211 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
1
vote
1answer
595 views

Should scaling be done for mixed data (categorical and numerical)?

My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations ...
1
vote
1answer
133 views

Transform test data when using a persistent model

I'm quite new to data science and only slowly following the necessary steps to get valid results using scikit-learn. As far as I understand you fit and transform the training data and only transform ...
1
vote
1answer
191 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
2
votes
2answers
214 views

How to Normalise features for small datasets?

I am working with a small dataset ( N = 50 ). I would like to normalise my input features. I am facing the following issues: Because of the small size of the dataset the range of training input ...
1
vote
2answers
553 views
3
votes
2answers
46 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
0answers
141 views

I have transformed my cyclic variables into sin-cos variables; will I need further normalization/standardization?

I have a dataset where there are both numerical, categorical and cyclic(month-quarter) variables. I will run a regression model, but I may also use Random Forest, XGBoost etc. So I will preprocess my ...
1
vote
1answer
20 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
1
vote
1answer
116 views

Feature selection before or after applying filter in Time-series forecasting

I'm predicting ozone concentration based on meteorological variables and ozone value of the previous day. I applied savitzky golay filter to get rid of noise in the time-series dataset. My question ...
3
votes
5answers
4k views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
1
vote
0answers
60 views

Sparsify meaningful data following a Gaussian distribution

Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the meaningful information, that is the information that lies far away from the mean. One way to do ...
2
votes
3answers
1k views

One Hot Label Encoding Scikit_learn convert back to Data Frame

I have a data frame with 4 features and 1 target. The 4 features are 3 categorical and 1 numerical. I created X which is a new data frame for the 3 categorical features. I use one hot label encoding ...
1
vote
0answers
11 views
2
votes
0answers
24 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
3
votes
2answers
5k views

How to export PCA to use in another program

I'm trying to write a random forest classifier for a very large dataset, as such as part of the pre-processing i have applied PCA to reduce from 643 features to 5 PC's. Is it possible to export these ...
2
votes
0answers
188 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
0
votes
0answers
197 views

Pre-processing data to make predictions on deployed Sklearn model

I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...) I have used pickle to ...
4
votes
2answers
6k views

Convert exponential to normal distribution

For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better ...
0
votes
1answer
5k views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
1
vote
0answers
35 views

Labeling audio dataset [duplicate]

I would like to try and create my own audio dataset which I can then use to train machine learning models for classification. The data that I've gathered consists of multiple long audio files of ...

1
4
5
6 7 8