Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Remove noise from a k dimensional dataset made up of connected components situated far away from each other

I want to remove "meaningless" points from a k-dimensional datset which's structure follows some predefined rules. In order to make the problem easier to visualize, I'll use a 2-d datset. Dataset ...
0
votes
1answer
19 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
0
votes
0answers
8 views

Error in the train function

I'm building a model and i want to re preocess the data using PCA but i have an error in the train function : ...
0
votes
0answers
61 views
0
votes
1answer
37 views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
17 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
0answers
17 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
0
votes
0answers
20 views

GPS Data Preprocessing Recommandations?

GPS Data Preprocessing I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
1answer
12 views

Should I have “normal” sampled data in my dataset?

I am busy working on a project to find the reasons why kids in normal households are doing badly in school. I have a dataset of which consists of kids that live in environments where the family is ...
0
votes
1answer
36 views

Reconstituting estimated/predicted values to original scale from MinMaxScaler

Thank you all, I am playing around with a deterministic function in order to understand machine learning as in the tutorial blog at https://machinelearningmastery.com/tutorial-first-neural-network-...
1
vote
0answers
19 views

How to treat Compass data in random forest regression

I'm working on a project where two of the features are entryHeading and exitHeading. Both state the direction (N, NE, E, SE, S, SW, W) of a vehicle at multiple points. My question is how would i go ...
0
votes
0answers
22 views

How to prepare coordinate sequence data for machine learning classification?

I want to perform a task where the goal is to classify coordinate sequences by labels. The raw data consists of temporal log sequences for each label like this: ...
0
votes
1answer
24 views

Transform test data when using a persistent model

I'm quite new to data science and only slowly following the necessary steps to get valid results using scikit-learn. As far as I understand you fit and transform the training data and only transform ...
2
votes
2answers
29 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
1
vote
1answer
45 views

Should scaling be done for mixed data (categorical and numerical)?

My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations ...
1
vote
1answer
25 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
1
vote
1answer
33 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm. I made sure that the class counts are balanced. i.e., an equal number of ...
0
votes
0answers
14 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
0
votes
0answers
17 views

Multi-label Image Classification Neural Network

I am attempting to complete an old Kaggle competition on the iMet collection but having difficulties understanding how to preprocess the data. The data set consists of photos with a photo_id, a csv ...
3
votes
2answers
43 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
1answer
20 views

Filling created feature with values

I'm trying to improve accuracy. I created a few new features based on old features. So I need to fill new feature's empty cell with same values in order to equaling shapes.Then, I tried it with median ...
1
vote
0answers
13 views

I have transformed my cyclic variables into sin-cos variables; will I need further normalization/standardization?

I have a dataset where there are both numerical, categorical and cyclic(month-quarter) variables. I will run a regression model, but I may also use Random Forest, XGBoost etc. So I will preprocess my ...
1
vote
1answer
11 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
2
votes
0answers
48 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
0
votes
1answer
29 views

Feature selection before or after applying filter in Time-series forecasting

I'm predicting ozone concentration based on meteorological variables and ozone value of the previous day. I applied savitzky golay filter to get rid of noise in the time-series dataset. My question ...
1
vote
0answers
46 views

Sparsify meaningful data following a Gaussian distribution

Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the meaningful information, that is the information that lies far away from the mean. One way to do ...
0
votes
0answers
8 views

prediction with un-obligatory features

I am pre-processing a real-world dataset with some features with missing values. these features are not mandatory - the user doesn't have to provide them. This means the values are not missing at ...
1
vote
0answers
9 views

fit transform with mysterious special chracters

I tried to make the bag of words ...
1
vote
2answers
49 views

How can I use mean normalization. Should I use it for numerical columns or categorical columns as well?

Should we normalize the categorical columns in our dataset? Or just the numerical columns?
1
vote
0answers
11 views

Preparing new samples for ML classifier with the same encoders

I was wondering what's the best way to preprocess new samples for my ML classifier. I have a raw data with about 3000 samples. I'm preprocessing it with some LabelEncoders and TargetEncdoders for ...
0
votes
0answers
13 views

Preprocessing data in image segmentation problem

I am implementing a research paper on image segmentation. Following are the image segmentation steps which are to be done before training its network- 1.Following image normalization is used- ...
2
votes
0answers
22 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
0
votes
1answer
64 views

Positively skewed target label in regression

I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-...
1
vote
2answers
53 views

How to export PCA to use in another program

I'm trying to write a random forest classifier for a very large dataset, as such as part of the pre-processing i have applied PCA to reduce from 643 features to 5 PC's. Is it possible to export these ...
1
vote
0answers
25 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
0
votes
1answer
26 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
0
votes
0answers
8 views

How would one reduce dimensionality/covariance in a dataset with nonlinearly covariant variables? Is M-SSA a no-go?

I am familiar with the Principal Component Analysis method of covariance and dimensionality reduction. I am considering using its multivariate time series brother, Multivariate Singular Spectrum ...
1
vote
3answers
112 views

One Hot Label Encoding Scikit_learn convert back to Data Frame

I have a data frame with 4 features and 1 target. The 4 features are 3 categorical and 1 numerical. I created X which is a new data frame for the 3 categorical features. I use one hot label encoding ...
0
votes
0answers
41 views

preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
0
votes
0answers
18 views

Handling collection of featurevectors for classification

I have a data set where devices are represented by a collection of variables. These variables consist of several properties like a name, datatype, driver, limit values, etc. (mixed data; quantitative ...
0
votes
0answers
40 views

Pre-processing data to make predictions on deployed Sklearn model

I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...) I have used pickle to ...
0
votes
0answers
19 views

Feature vector of linear model

I read this paper that applies logistic regression to a dataset generated from a simulation they created. The dataset contains a set of binary vectors (called challenges) that looks like this: ...
0
votes
0answers
19 views

ZCA: Covariance of Large Image Database

I have an overall question if my method is sound, so please bear with me in this description :). I have a large image database and I wanted to create a preprocessing step using ZCA. The issue is that ...
0
votes
2answers
163 views

sklearn.preprocessing.MinMaxScaler: non-broadcastable output shape error

Why I am getting the following error: ValueError: non-broadcastable output operand with shape (1,1) doesn't match the broadcast shape (1,2) While executing: ...
0
votes
1answer
552 views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
0
votes
0answers
16 views

How to avoid the alternating conditional expectations from transforming most of the response data to very close values?

I am applying alternating conditional expectations (ACE) in a forward stepwise manner, similarly to the authors of the original ACE paper. My dataset has 103 predictor variables $x_i$, one response ...
2
votes
5answers
138 views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
1
vote
1answer
23 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
24 views

Labeling audio dataset [duplicate]

I would like to try and create my own audio dataset which I can then use to train machine learning models for classification. The data that I've gathered consists of multiple long audio files of ...
0
votes
1answer
32 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...