A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
34 views

Can anyone direct me on how to got to [closed]

I am a newbie to python. I have uploaded about 2000 images into my google drive, assigned a variable to all content in the folder and tried to split the folder's content into training and testing, ...
1
vote
0answers
12 views

Algorithm for EMG dataset

I have an EMG dataset that depicts 6 different gestures depending on the measurements at intervals of roughly 1ms, from 5 electrodes placed equidistant on the wrist. I have the data for 36 different ...
1
vote
0answers
13 views

Replace values before and after a certain value [closed]

I have a variable like this: V1: 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 I want to change the 0 before the 1 to 1, and the 0 after the 2 to 2. It is supposed to look like this: V1: 1 1 1 1 1 1 1 0 0 0 0 ...
3
votes
1answer
73 views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
2
votes
1answer
33 views

How can I perform categorical encoding when the dataset is too large for memory?

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
1
vote
1answer
40 views

For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
2
votes
1answer
23 views

Rescaling each image Individually with keras

I am a beginner working on a simple CNN to classify X-ray detector images. Due to source intensity, all images have different max values. I want to use ImageDataGenerator to rescale those images to be ...
3
votes
1answer
64 views

ASR on low dataset

I am doing an ASR(automatic speech recognition) as master thesis on low key dataset. Voice and text data is labelled. There are around 4000 phrases and around 5 hours speech. I don't have background ...
2
votes
1answer
54 views

Format of data in SQL for machine learning

I am a beginner at Machine Learning and am starting out on a ML project. I have a large chunk of the source material and have started extracting the data from it to be stored in SQL (initial test with ...
1
vote
1answer
31 views

One scaler for all features or one scaler per feature?

I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?
1
vote
2answers
48 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
2
votes
1answer
29 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
0
votes
0answers
12 views

How does setting preProcess argument in train function in Caret work?

I am trying to predict the times table training a neural network. However, I couldn't really get how preProcess argument works in ...
1
vote
1answer
18 views

On which step should use SMOTE technique for over sampling?

I want to use SMOTE technique for over sampling but I don't know on which step on pre-processing I should use it. My preprocessing steps are: Missing values Removing Outliers Smoothing Data Should ...
1
vote
0answers
12 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
0
votes
0answers
28 views

Customize loss function for Music Generation LSTM (?)

I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract ...
1
vote
3answers
64 views

Correlation between categorical variables based on the target distribution

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
1
vote
3answers
52 views

How to handle “year” variable for Machine Learning models

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable? Thanks ...
1
vote
0answers
12 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
1answer
15 views

What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what ...
0
votes
0answers
20 views

Split time series data into Train Test and Valid sets in Python

I'm working on a project in which I have combined 2 datasets if time series (e.g D1, D2). D1 was with the 5-minutes interval and ...
1
vote
1answer
30 views

Improving the performace of the Naive Bayes classifier by decorrelating the data

I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $...
0
votes
1answer
28 views

Preprocessing of 3D CAD files for Keras Conv3D input

I'd like to apply some machine learning on 3D CAD data. File format should ideally be mesh-based like STL. Keras offers 3D convolutional layers (https://keras.io/layers/convolutional/), so it can ...
0
votes
1answer
15 views

How much of a disadvantage is a small sample size?

I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my ...
1
vote
0answers
34 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
1
vote
0answers
20 views

Column of Lists into Regex vs. Tuple

I'm working on preparing/analysing this dataset of Russian propaganda ads, in which there are columns which contain lists. 'interests_categories', for example, has a row which contains ['Unknown', '...
1
vote
2answers
28 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
0
votes
0answers
155 views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
0
votes
0answers
18 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
2
votes
1answer
43 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
1
vote
0answers
10 views

Handling a large dataset consisting of npy files

I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load ...
1
vote
0answers
24 views

How to label time series data correctly for training RNN/CNN models?

My Case I want to tackle a deep learning classification task using various smartphone sensor data. I will use a self-built data acquisition app and basically walk around with the phone, manually ...
0
votes
0answers
8 views

Suitable aggregating method for condensing sets of binary values into a single real one

I have a dataset composed of several subjects. Each subject has a series of binary indicators where 1 indicates that a the subject presents an indicator and 0 means that the indicator is not present. ...
0
votes
0answers
72 views

Suggestion for handling specific missing data

I have data, that describes distance from given location to nearest object (e.g. school, shop etc). Because of performance reasons I couldn't scrape the data about objects, that are futher away than 2....
0
votes
0answers
4 views

Remove noise from a k dimensional dataset made up of connected components situated far away from each other

I want to remove "meaningless" points from a k-dimensional datset which's structure follows some predefined rules. In order to make the problem easier to visualize, I'll use a 2-d datset. Dataset ...
0
votes
1answer
89 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
0
votes
0answers
11 views

Error in the train function

I'm building a model and i want to re preocess the data using PCA but i have an error in the train function : ...
0
votes
1answer
255 views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
28 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
0answers
23 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
0
votes
0answers
24 views

GPS Data Preprocessing Recommandations?

GPS Data Preprocessing I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
1answer
12 views

Should I have “normal” sampled data in my dataset?

I am busy working on a project to find the reasons why kids in normal households are doing badly in school. I have a dataset of which consists of kids that live in environments where the family is ...
1
vote
1answer
76 views

Reconstituting estimated/predicted values to original scale from MinMaxScaler

Thank you all, I am playing around with a deterministic function in order to understand machine learning as in the tutorial blog at https://machinelearningmastery.com/tutorial-first-neural-network-...
1
vote
0answers
20 views

How to treat Compass data in random forest regression

I'm working on a project where two of the features are entryHeading and exitHeading. Both state the direction (N, NE, E, SE, S, SW, W) of a vehicle at multiple points. My question is how would i go ...
0
votes
0answers
64 views

How to prepare coordinate sequence data for machine learning classification?

I want to perform a task where the goal is to classify coordinate sequences by labels. The raw data consists of temporal log sequences for each label like this: ...
0
votes
1answer
61 views

Transform test data when using a persistent model

I'm quite new to data science and only slowly following the necessary steps to get valid results using scikit-learn. As far as I understand you fit and transform the training data and only transform ...
2
votes
2answers
36 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
1
vote
1answer
69 views

Should scaling be done for mixed data (categorical and numerical)?

My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations ...
1
vote
1answer
30 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
1
vote
1answer
39 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm. I made sure that the class counts are balanced. i.e., an equal number of ...