Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

117 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
1answer
18 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
3
votes
0answers
22 views

preprocessing time sequence

I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event. The preprocessing steps i took are: (1) turning to OneHotEncoding using pandas: <...
2
votes
1answer
27 views

How to use sklean pipeline to deal with data that read in line by line

The problem I'm facing is that my data is too big, i can't load it to a dataframe and then process it. However, I really want to use the sklearn pipeline API, so that I can reuse those subclass ...
2
votes
0answers
57 views

Will flattening multivariate time series data before clustering make the results meaningless?

I have a large number of financial time series that I wish to do cluster analysis on. Each time series has the same length and spans multiple years of daily data (returns, volatility, etc.). As part ...
2
votes
0answers
50 views

How to label time series data correctly for training RNN/CNN models?

My Case I want to tackle a deep learning classification task using various smartphone sensor data. I will use a self-built data acquisition app and basically walk around with the phone, manually ...
2
votes
1answer
91 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
2
votes
0answers
22 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
2
votes
0answers
820 views

One hot encoding as input to recurrent neural networks

I'm trying to predict next label in a pattern based on previous labels using recurrent neural network. In total I have 100 labels Example of input pattern: ...
2
votes
0answers
162 views

Audio files and their corresponding spectrograms for image classification process

Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding ...
2
votes
1answer
122 views

Clustering time series based on monotonic similarity

Context I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, ...
2
votes
2answers
608 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
2
votes
2answers
237 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
2
votes
1answer
468 views

normalization/denormalization for linear regression problem

My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after ...
2
votes
0answers
171 views

how is countvectorizer used in real production environment?

how is countvectorizer used in real production environment? do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?...
2
votes
0answers
303 views

User activity representation for Prediction/ML

I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data. The raw data is a sequence of all activities by all users: ...
1
vote
0answers
15 views

Split-Half Reliability Python

I have a pandas dataframe of baseball stats that I have transposed that looks as follows: Each column represents a separate plate appearance for a player. Each player can (and usually is) represented ...
1
vote
0answers
18 views

How to handle negative delays when predicting flight delays?

I am working with the nycflights dataset. My goal is to predict departure and arrival time delays using random forests. My ...
1
vote
1answer
21 views

should we include or exclude a variable in a logistic regression based on the description below?

should we include or exclude a variable in a logit regr. model which will only obtain values if a certain event takes place otherwise will show N/A? this variable tells whether or not a product will ...
1
vote
0answers
12 views

Forecast multiple unevenly spaced time series

I am building a time-series forecasting model to predict some patterns in climatological data. The dataset consists of many (2 mln) time series which look for example as: However the observations ...
1
vote
0answers
23 views

svd to reduce text matricies

I am having problems with very big texts. after preprocessing each of my docs represented as a matrix of sentences, where each sentence represented as encoded words (each word have a unique vocab ...
1
vote
1answer
40 views

Forecasting with a Machine Learning Algorithm

Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all ...
1
vote
0answers
22 views

MinMaxScaling vs L1/L2-Normalization

I'm wondering about the difference or the application of the different types of rescaling data. So far, I'm aware that standardization assumes the data has a gaussian distribution. So if this is the ...
1
vote
0answers
19 views

Algorithm for EMG dataset

I have an EMG dataset that depicts 6 different gestures depending on the measurements at intervals of roughly 1ms, from 5 electrodes placed equidistant on the wrist. I have the data for 36 different ...
1
vote
0answers
13 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
1
vote
1answer
52 views

Customize loss function for Music Generation LSTM (?)

I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract ...
1
vote
0answers
15 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
1
vote
0answers
62 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
1
vote
0answers
21 views

Column of Lists into Regex vs. Tuple

I'm working on preparing/analysing this dataset of Russian propaganda ads, in which there are columns which contain lists. 'interests_categories', for example, has a row which contains ['Unknown', '...
1
vote
0answers
50 views

Handling a large dataset consisting of npy files

I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load ...
1
vote
0answers
38 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
1
vote
1answer
150 views

Reconstituting estimated/predicted values to original scale from MinMaxScaler

Thank you all, I am playing around with a deterministic function in order to understand machine learning as in the tutorial blog at https://machinelearningmastery.com/tutorial-first-neural-network-...
1
vote
1answer
30 views

How to treat Compass data in random forest regression

I'm working on a project where two of the features are entryHeading and exitHeading. Both state the direction (N, NE, E, SE, S, SW, W) of a vehicle at multiple points. My question is how would i go ...
1
vote
1answer
27 views

Filling created feature with values

I'm trying to improve accuracy. I created a few new features based on old features. So I need to fill new feature's empty cell with same values in order to equaling shapes.Then, I tried it with median ...
1
vote
0answers
45 views

I have transformed my cyclic variables into sin-cos variables; will I need further normalization/standardization?

I have a dataset where there are both numerical, categorical and cyclic(month-quarter) variables. I will run a regression model, but I may also use Random Forest, XGBoost etc. So I will preprocess my ...
1
vote
0answers
48 views

Sparsify meaningful data following a Gaussian distribution

Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the meaningful information, that is the information that lies far away from the mean. One way to do ...
1
vote
0answers
10 views

fit transform with mysterious special chracters

I tried to make the bag of words ...
1
vote
0answers
12 views

Preparing new samples for ML classifier with the same encoders

I was wondering what's the best way to preprocess new samples for my ML classifier. I have a raw data with about 3000 samples. I'm preprocessing it with some LabelEncoders and TargetEncdoders for ...
1
vote
0answers
101 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
1
vote
0answers
86 views

preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
1
vote
1answer
76 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
80 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
1
vote
0answers
23 views

How can I transform (pre-process) pure count data for PCA analysis?

If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?
1
vote
1answer
171 views

Is there any NLP library or package which can help in adding comma, punctuation, newlines appropriately to text?

I have a movie transcript, without commas, punctuation or newlines. Is there any NLP technique which can help to implement this?
1
vote
0answers
21 views

Group data without losing information

Context Imagine that I have a dataset about sending messages. Each row as user_id, a batch_id, a ...
1
vote
0answers
63 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
1
vote
0answers
435 views

Decision Tree - Preprocessing for very sparse features

How do we pre-process data for very sparse features for a decision tree? From this Turi documentation for decision trees It mentioned this: Why chose decision trees? Different kinds of models ...
1
vote
0answers
108 views

Image preprocessing: How to resize / align / cut images of various sizes?

I want to create a new dataset for image recognition. If I have Object A that I want to recognize in images, and multiple images of various different sizes, how do I preprocess them so that they ...
1
vote
0answers
240 views

Should I remove the trend from timeseries when using DeepAR

I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM) I figured out from the paper that SageMaker's ...
1
vote
0answers
57 views

Construction Adjacency matrix for GNN

A bit of a novice question, my data comes in the form of 2D hit points, my goal is to perform node/edge classification to find out whether this graph is my signal or background. My question is, when ...
1
vote
0answers
27 views

Specific data formatting techniques for discontiguous time series?

I'm facing a predicting problem for food alerts. The goal is to predict the variables of the most probable alert in the next x days (also any information I could get about future alerts is really ...