Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
6
votes
5answers
1k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
0answers
8 views

Given a dataset how do i determine the number of modes in the underlying distribution?

given some data is there a technique determine the number of modes (Unimodal distribution, bimodal distribution etc.) without plotting and visualization?
3
votes
1answer
36 views

Iterate over multiple dataframe rows at the same time

I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values. I need to loop over ...
0
votes
1answer
12 views

How do you choose the min and max for the min-max normalization on a histogram classifier?

Please let me know what to do when there is a value in the testing set is bigger than the max value used to min-max normalize the training set building a histogram classifier. Do I go back and change ...
1
vote
0answers
21 views

svd to reduce text matricies

I am having problems with very big texts. after preprocessing each of my docs represented as a matrix of sentences, where each sentence represented as encoded words (each word have a unique vocab ...
0
votes
0answers
33 views

How to correctly manage predictions when the inputs are outbound the original scaling range?

I have a neural network for a regression problem that was trained using MinMaxScaler(0,1) for features and I have two questions with this. I often find that scaling the output (or target variable) ...
0
votes
0answers
16 views

How to use Predefined Split for Randomized SearchCV

I'm trying to regularize my random forest regressor with RandomizedSearchCV. With RandomizedSearchCV the train and test are not ...
1
vote
2answers
21 views

Pre-processing mixed data prior to clustering

I am new to hierarchical clustering, and wish to perform clustering on mixed data. I am slightly confused on the necessary pre-processing steps. I understand how to pre-process purely continuous data, ...
0
votes
1answer
23 views

Dimensionality reduction without select components

I would like to use dimensionality reduction algorithm in my pipeline. I have 2k features and I'm using xgboost. My model is rebuilding each day (there are new records that should be involve to ...
1
vote
1answer
23 views

Representing 2 dimensional data for facial recognition

My goal is to train a neural network to recognize faces based on a list of landmark points generated by Google Firebase ML-Kit. Since I just started familiarizing myself with ML, I only want my model ...
0
votes
1answer
23 views

Should we first remove the unwanted independent variable or split the data into test train data?

I wanted to know if we are dropping any independent variable as it has too many missing values(~75% or more) from the data then should we do it before splitting the data or after splitting the data
2
votes
0answers
20 views

preprocessing time sequence

I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event. The preprocessing steps i took are: (1) turning to OneHotEncoding using pandas: <...
0
votes
1answer
42 views

I have a pandas dataframe and i need to clear all the special characters other than space

Input: import pandas as pd df=pd.read_excel("OCRFinal.xlsx") df['OCR_Text']=df['OCR_Text'].str.replace(r'\W+'," ") print(df['OCR_Text']) Output: The excel ...
1
vote
1answer
39 views

Forecasting with a Machine Learning Algorithm

Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all ...
1
vote
0answers
21 views

MinMaxScaling vs L1/L2-Normalization

I'm wondering about the difference or the application of the different types of rescaling data. So far, I'm aware that standardization assumes the data has a gaussian distribution. So if this is the ...
0
votes
1answer
110 views

How to preprocess data for Word2Vec?

I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?
0
votes
0answers
14 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
2
votes
1answer
37 views

Word Embedding or Hash?

In my dataset I have a 'text' column and a 'followers' column containing lists of follower IDs, i.e. '1093777852477116417, 936194589043683328,...'. Some of the 'followers' values contain thousands of ...
2
votes
0answers
41 views

Will flattening multivariate time series data before clustering make the results meaningless?

I have a large number of financial time series that I wish to do cluster analysis on. Each time series has the same length and spans multiple years of daily data (returns, volatility, etc.). As part ...
1
vote
0answers
37 views

Can anyone direct me on how to got to [closed]

I am a newbie to python. I have uploaded about 2000 images into my google drive, assigned a variable to all content in the folder and tried to split the folder's content into training and testing, ...
1
vote
0answers
17 views

Algorithm for EMG dataset

I have an EMG dataset that depicts 6 different gestures depending on the measurements at intervals of roughly 1ms, from 5 electrodes placed equidistant on the wrist. I have the data for 36 different ...
1
vote
0answers
16 views

Replace values before and after a certain value [closed]

I have a variable like this: V1: 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 I want to change the 0 before the 1 to 1, and the 0 after the 2 to 2. It is supposed to look like this: V1: 1 1 1 1 1 1 1 0 0 0 0 ...
3
votes
1answer
258 views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
2
votes
2answers
42 views

How can I perform categorical encoding when the dataset is too large for memory?

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
1
vote
1answer
67 views

For outliers treatment, clipping, winsorizing or removing?

I came across two different techniques for treating outliers winsorization, clipping and removing: Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, ...
2
votes
1answer
38 views

Rescaling each image Individually with keras

I am a beginner working on a simple CNN to classify X-ray detector images. Due to source intensity, all images have different max values. I want to use ImageDataGenerator to rescale those images to be ...
3
votes
1answer
80 views

ASR on low dataset

I am doing an ASR(automatic speech recognition) as master thesis on low key dataset. Voice and text data is labelled. There are around 4000 phrases and around 5 hours speech. I don't have background ...
2
votes
1answer
59 views

Format of data in SQL for machine learning

I am a beginner at Machine Learning and am starting out on a ML project. I have a large chunk of the source material and have started extracting the data from it to be stored in SQL (initial test with ...
1
vote
1answer
33 views

One scaler for all features or one scaler per feature?

I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?
1
vote
2answers
75 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
2
votes
1answer
29 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
0
votes
0answers
27 views

How does setting preProcess argument in train function in Caret work?

I am trying to predict the times table training a neural network. However, I couldn't really get how preProcess argument works in ...
1
vote
1answer
23 views

On which step should use SMOTE technique for over sampling?

I want to use SMOTE technique for over sampling but I don't know on which step on pre-processing I should use it. My preprocessing steps are: Missing values Removing Outliers Smoothing Data Should ...
1
vote
0answers
12 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
0
votes
0answers
37 views

Customize loss function for Music Generation LSTM (?)

I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract ...
1
vote
3answers
88 views

Correlation between categorical variables based on the target distribution

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
1
vote
3answers
88 views

How to handle “year” variable for Machine Learning models

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable? Thanks ...
1
vote
0answers
14 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
0
votes
2answers
33 views

What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what ...
0
votes
0answers
68 views

Split time series data into Train Test and Valid sets in Python

I'm working on a project in which I have combined 2 datasets if time series (e.g D1, D2). D1 was with the 5-minutes interval and ...
1
vote
1answer
44 views

Improving the performace of the Naive Bayes classifier by decorrelating the data

I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $...
0
votes
2answers
58 views

Preprocessing of 3D CAD files for Keras Conv3D input

I'd like to apply some machine learning on 3D CAD data. File format should ideally be mesh-based like STL. Keras offers 3D convolutional layers (https://keras.io/layers/convolutional/), so it can ...
0
votes
1answer
18 views

How much of a disadvantage is a small sample size?

I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my ...
1
vote
0answers
52 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
1
vote
0answers
20 views

Column of Lists into Regex vs. Tuple

I'm working on preparing/analysing this dataset of Russian propaganda ads, in which there are columns which contain lists. 'interests_categories', for example, has a row which contains ['Unknown', '...
1
vote
2answers
31 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
2
votes
0answers
653 views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
0
votes
0answers
19 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
2
votes
1answer
55 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
1
vote
0answers
21 views

Handling a large dataset consisting of npy files

I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load ...

1
2 3 4 5 6