Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

128 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
1answer
52 views

How to use sklean pipeline to deal with data that read in line by line

The problem I'm facing is that my data is too big, i can't load it to a dataframe and then process it. However, I really want to use the sklearn pipeline API, so that I can reuse those subclass ...
3
votes
1answer
38 views

preprocessing time sequence

I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event. The preprocessing steps i took are: (1) turning to OneHotEncoding using pandas: <...
3
votes
0answers
1k views

One hot encoding as input to recurrent neural networks

I'm trying to predict next label in a pattern based on previous labels using recurrent neural network. In total I have 100 labels Example of input pattern: ...
3
votes
2answers
901 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
3
votes
3answers
149 views

Is there any tool for data visualization and manipulation?

I have a time series data set that I need to manually label for supervised learning. What I am doing now is using excel to the plot, and when I see the pattern that I want, I hover over the data on ...
2
votes
2answers
90 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
2
votes
1answer
63 views

How to preprocess an ordered categorical variable to feed a machine learning algorithm?

I have a categorical variable that measures the income of a family: A: no income B: Up to $500 C: $500-$700 … P: $5000-$6000 Q: More than \\\$6000 It seems odd to ...
2
votes
0answers
58 views

Forecast multiple unevenly spaced time series

I am building a time-series forecasting model to predict some patterns in climatological data. The dataset consists of many (2 mln) time series which look for example as: However the observations ...
2
votes
0answers
182 views

Will flattening multivariate time series data before clustering make the results meaningless?

I have a large number of financial time series that I wish to do cluster analysis on. Each time series has the same length and spans multiple years of daily data (returns, volatility, etc.). As part ...
2
votes
0answers
125 views

How to label time series data correctly for training RNN/CNN models?

My Case I want to tackle a deep learning classification task using various smartphone sensor data. I will use a self-built data acquisition app and basically walk around with the phone, manually ...
2
votes
0answers
24 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
2
votes
1answer
416 views

Positively skewed target label in regression

I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-...
2
votes
0answers
188 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
2
votes
0answers
185 views

Audio files and their corresponding spectrograms for image classification process

Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding ...
2
votes
1answer
147 views

Clustering time series based on monotonic similarity

Context I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, ...
2
votes
2answers
498 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
2
votes
2answers
741 views

normalization/denormalization for linear regression problem

My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after ...
2
votes
0answers
177 views

how is countvectorizer used in real production environment?

how is countvectorizer used in real production environment? do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?...
2
votes
0answers
314 views

User activity representation for Prediction/ML

I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data. The raw data is a sequence of all activities by all users: ...
1
vote
0answers
7 views

Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
1
vote
1answer
8 views

Determining direction from a series of bounding boxes

I have a sequence of bounding boxes for people moving in and out of a camera's view. I'd like to determine the direction each person is moving (basically just left/right), so I can apply this as a ...
1
vote
1answer
21 views

Small object detection preprocessing

I am currently working on an object detection project in which I amtrying to detect very small objects 50x50 object in a 2k image. EfficientDet produced a very low result if I just put the raw ...
1
vote
0answers
25 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
1
vote
0answers
83 views

Sliding Window Normalization

I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize ...
1
vote
0answers
28 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
1
vote
1answer
21 views

Quality check for preprocessing of Text data

I have developed a pipeline for text data preprocessing with different clean up techniques like Stemming , Lemmatization, Stop words removal etc. But now the ask from the business team is to quantify ...
1
vote
1answer
28 views

Log scaling whole dataset

I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value. I am wondering how I can get rid of this -infinity? I have been advised to add a small value ...
1
vote
1answer
64 views

What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
1
vote
0answers
10 views
1
vote
0answers
11 views

Preparing training data for crop species classification from drone images

I have high resolution multispectral (R,G,B,NIR,RE bands) images of a field taken from MicaSense RedEdge mounted on a drone. There are various species of crops planted. I want to classify the crops or ...
1
vote
0answers
14 views

Redo preprocess on unknown row

I'm trying to write a script to get the most similar rows in a certain dataframe, based on a single row. Using scikit-learn The method I need is ...
1
vote
2answers
43 views

How to combine two time dependent datasets?

I'm very new to data science so please be gentle. I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one ...
1
vote
1answer
62 views

Data scaling for large dynamic range in neural networks

The usual strategy in neural networks today is to use min-max scaling to scale the input feature vector from 0 to 1. I want to know if the same principle holds true if our inputs have a large dynamic ...
1
vote
0answers
77 views

How to handle negative delays when predicting flight delays?

I am working with the nycflights dataset. My goal is to predict departure and arrival time delays using random forests. My ...
1
vote
0answers
15 views

Is there any problem using Euclidean Distance for a data set that includes a lot of values between 0 and 1?

I have a data set which includes 3 types of financial data (P/E, ROA, EPS growth) for each stock in a list of stocks. My goal is to use K-Medoids to cluster these stocks into groups based on this data....
1
vote
1answer
59 views

How do you choose the min and max for the min-max normalization on a histogram classifier?

Please let me know what to do when there is a value in the testing set is bigger than the max value used to min-max normalize the training set building a histogram classifier. Do I go back and change ...
1
vote
0answers
25 views

svd to reduce text matricies

I am having problems with very big texts. after preprocessing each of my docs represented as a matrix of sentences, where each sentence represented as encoded words (each word have a unique vocab ...
1
vote
1answer
55 views

Forecasting with a Machine Learning Algorithm

Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all ...
1
vote
0answers
86 views

MinMaxScaling vs L1/L2-Normalization

I'm wondering about the difference or the application of the different types of rescaling data. So far, I'm aware that standardization assumes the data has a gaussian distribution. So if this is the ...
1
vote
1answer
20 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
1
vote
0answers
14 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
1
vote
1answer
142 views

Customize loss function for Music Generation LSTM (?)

I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract ...
1
vote
0answers
20 views

Creating Flags Instead of Designated Values

I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment....
1
vote
0answers
112 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
1
vote
0answers
22 views

Column of Lists into Regex vs. Tuple

I'm working on preparing/analysing this dataset of Russian propaganda ads, in which there are columns which contain lists. 'interests_categories', for example, has a row which contains ['Unknown', '...
1
vote
1answer
117 views

Handling a large dataset consisting of npy files

I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load ...
1
vote
1answer
104 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
1
vote
1answer
6k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
1
vote
0answers
135 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
0answers
41 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...