Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
5 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
2
votes
0answers
12 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
0
votes
1answer
72 views

what is correct way to perform normalization on data in Auto encoder?

working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of ...
2
votes
1answer
315 views

normalization/denormalization for linear regression problem

My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after ...
0
votes
1answer
24 views

Feature selection before or after applying filter in Time-series forecasting

I'm predicting ozone concentration based on meteorological variables and ozone value of the previous day. I applied savitzky golay filter to get rid of noise in the time-series dataset. My question ...
2
votes
2answers
126 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
0
votes
1answer
29 views

Positively skewed target label in regression

I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-...
2
votes
5answers
76 views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
0
votes
1answer
36 views

How to deal with name strings in large data sets for ML?

My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later. Some word embedding techniques were used for longer text ...
1
vote
0answers
35 views

Sparsify meaningful data following a Gaussian distribution

Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the meaningful information, that is the information that lies far away from the mean. One way to do ...
0
votes
0answers
8 views

prediction with un-obligatory features

I am pre-processing a real-world dataset with some features with missing values. these features are not mandatory - the user doesn't have to provide them. This means the values are not missing at ...
0
votes
1answer
35 views

How to discretize certain features with a feature set?

I am working with typing data with timing features(unit: ms) and some of the features are based on the keyboard keyCodes(positive integers, range:[8, 222]). Currently, I use ...
1
vote
3answers
56 views

One Hot Label Encoding Scikit_learn convert back to Data Frame

I have a data frame with 4 features and 1 target. The 4 features are 3 categorical and 1 numerical. I created X which is a new data frame for the 3 categorical features. I use one hot label encoding ...
1
vote
2answers
32 views

How can I use mean normalization. Should I use it for numerical columns or categorical columns as well?

Should we normalize the categorical columns in our dataset? Or just the numerical columns?
1
vote
0answers
7 views
1
vote
2answers
51 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
2
votes
1answer
64 views

Clustering time series based on monotonic similarity

Context I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, ...
1
vote
0answers
10 views

Preparing new samples for ML classifier with the same encoders

I was wondering what's the best way to preprocess new samples for my ML classifier. I have a raw data with about 3000 samples. I'm preprocessing it with some LabelEncoders and TargetEncdoders for ...
2
votes
2answers
2k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
0
votes
2answers
78 views

sklearn.preprocessing.MinMaxScaler: non-broadcastable output shape error

Why I am getting the following error: ValueError: non-broadcastable output operand with shape (1,1) doesn't match the broadcast shape (1,2) While executing: ...
0
votes
0answers
10 views

Preprocessing data in image segmentation problem

I am implementing a research paper on image segmentation. Following are the image segmentation steps which are to be done before training its network- 1.Following image normalization is used- ...
2
votes
0answers
21 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
1
vote
2answers
29 views

How to export PCA to use in another program

I'm trying to write a random forest classifier for a very large dataset, as such as part of the pre-processing i have applied PCA to reduce from 643 features to 5 PC's. Is it possible to export these ...
1
vote
0answers
21 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
1
vote
1answer
21 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
0
votes
1answer
21 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
0
votes
1answer
31 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
0
votes
0answers
8 views

How would one reduce dimensionality/covariance in a dataset with nonlinearly covariant variables? Is M-SSA a no-go?

I am familiar with the Principal Component Analysis method of covariance and dimensionality reduction. I am considering using its multivariate time series brother, Multivariate Singular Spectrum ...
0
votes
0answers
30 views

preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
0
votes
0answers
16 views

Handling collection of featurevectors for classification

I have a data set where devices are represented by a collection of variables. These variables consist of several properties like a name, datatype, driver, limit values, etc. (mixed data; quantitative ...
0
votes
0answers
31 views

Pre-processing data to make predictions on deployed Sklearn model

I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...) I have used pickle to ...
0
votes
0answers
18 views

Feature vector of linear model

I read this paper that applies logistic regression to a dataset generated from a simulation they created. The dataset contains a set of binary vectors (called challenges) that looks like this: ...
2
votes
2answers
2k views

Convert exponential to normal distribution

For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better ...
2
votes
2answers
5k views

One Hot Encoding vs Word Embeding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
0
votes
0answers
17 views

ZCA: Covariance of Large Image Database

I have an overall question if my method is sound, so please bear with me in this description :). I have a large image database and I wanted to create a preprocessing step using ZCA. The issue is that ...
0
votes
1answer
187 views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
15
votes
2answers
7k views

StandardScaler before and after splitting data

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
0
votes
0answers
10 views

How to avoid the alternating conditional expectations from transforming most of the response data to very close values?

I am applying alternating conditional expectations (ACE) in a forward stepwise manner, similarly to the authors of the original ACE paper. My dataset has 103 predictor variables $x_i$, one response ...
1
vote
0answers
17 views

Labeling audio dataset

I would like to try and create my own audio dataset which I can then use to train machine learning models for classification. The data that I've gathered consists of multiple long audio files of ...
0
votes
2answers
64 views

Understanding missing values in dataset

I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) ...
4
votes
5answers
6k views

issue with oneHotEncoding

So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course ...
0
votes
1answer
141 views

Different approaches of creating the test set

I came across different approaches to creating a test set. Theoretically, it's quite simple, just pick some instances randomly, typically 20% of the dataset and set them aside. Below are the ...
4
votes
1answer
76 views

Feature importance over a subset of instance space instead of an entire instance space

I'm really curious if anyone has faced this problem before, or is it even widely studied at all. Imagine we have a feature that isn't important (based on many widely available and textbook feature ...
0
votes
0answers
5 views

Class-inflated factor. At what point do we decide to remove it?

If we have a dataset with a few string-type factors that have a lot of one class, at what point do we decide to remove the said class? This question just came to mind since I was practicing on a ...
0
votes
1answer
27 views

Suggestion on Preprocessing dataset

I am trying to preprocess my dataset and needs some suggestion on it. The training data shape is : (166573, 14) The distribution of features : As you can see, only the first 4 columns go to ...
3
votes
4answers
362 views

When should ordinal data be represented catigorically and when as integer?

I am doing the Kaggle competition House Prices: Advanced Regression Techniques to learn more about data analysis. I would like to apply multiple models to the data(Regularized LR, Random Forests, ...
1
vote
1answer
37 views

Scaling features separately?

I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship ...
1
vote
0answers
43 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
0
votes
1answer
18 views

One attribute includes another attribute

I have a telecom dataset that has many attributes, among these attributes, there is "Voice mail plan" attribute that takes yes or no, and another attribute is "voice mail calls" which has many values, ...
1
vote
1answer
2k views

What does normalizing and mean centering data do?

Are there any concerns to normalizing data to be within the range 0 - 1 and mean centering the data as well? Does it matter which comes first? If you do one, is the other not required?