Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0
votes
1answer
13 views

How should a stateless data transformation be applied in regard to train/test split?

I want to apply spatial sign transformation to my data, but unlike other transformations this one is stateless. I am using sklearn and normallly i would first use ...
0
votes
1answer
8 views

How can I processing subgroup data [closed]

I am new to the Python, I have a data processing questions, I have a data frame like this: So as you could see there are 4 groups based on the group ID. What I want is in each group, in the last ...
0
votes
0answers
19 views

Data preprocessing - Time Series data resets its values - Detection & Correction

1. Summarize the problem I currently trying to work with time series data from sensors which has some problems regarding resetting it values. For example some cumulative values gets reset and don't ...
1
vote
1answer
8 views

Determining direction from a series of bounding boxes

I have a sequence of bounding boxes for people moving in and out of a camera's view. I'd like to determine the direction each person is moving (basically just left/right), so I can apply this as a ...
0
votes
1answer
25 views

Preprocess multi-sample time series data: encode each sample separately or in aggregate?

Let's say I have 3 dense sequences of uniform length. Should I fit a scaler on them separately or together? ...
0
votes
0answers
12 views

Inputting Unnormalized Data into Pretrained Resnet

Quick rookie question, I was wondering, in theory, what would happen if I input unscaled and unstandardized byte image data into a pretrained Resnet (namely, from Pytorch's library). Would it still ...
2
votes
2answers
48 views

How to handle non ordinal Features like Gender,Language,Region etc? Ordinal Encoding or one-hot encoding?

I see that usually, while preparing the dataset. Usually, data scientists convert non-ordinal features like Gender or Language in a dataset using LabelEncoder/ordinalEncoder. Ideally, they should have ...
0
votes
1answer
26 views

in binary classification where class labels are {-1, 1} is preprocessing needed?

In machine learning we convert labels using LabelEncoder to convert string ex:{"malignant", "benign"} -> {0, 1} I am wondering if converting Labels to any other numbers matter, ...
0
votes
0answers
22 views

Input pipeline with an autoencoder and tf.data

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...
0
votes
1answer
11 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
1
vote
1answer
21 views

Small object detection preprocessing

I am currently working on an object detection project in which I amtrying to detect very small objects 50x50 object in a 2k image. EfficientDet produced a very low result if I just put the raw ...
0
votes
0answers
19 views

How to fill missing latitude and longitude values in time series data?

I have time-series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
1
vote
2answers
35 views

How to convert longitude and latitude in time series data from daily to weekly?

I have time series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
0
votes
0answers
30 views

Predict on new data / Model Deployment

I do not undestand, how the deployment of a ML-model works in the reality. A given dataset needs to be mostly time pre-processed (for example One Hot Encoding). After will be a model cretead and ...
1
vote
1answer
15 views
0
votes
1answer
15 views

Storing and loading bottleneck features for transfer learning on large data sets (Keras)

I would like to apply transfer learning on a pretty large image data set in order to solve a classification problem. Currently I load a pre-trained net without the top layers, add my own top layers, ...
1
vote
1answer
55 views

Can someone clarify what the linear assumption of PCA is?

0 For the past few hours I've been trying to search what this linear assumption is. Some of the articles states that that your independent variables have to be linear in relationship and need some ...
0
votes
1answer
10 views

Treat similar observations in a classification problem

I have a dataset of about 200k rows and I'm working on a classification problem. Grouping the dataset by a key variable, I noticed that some rows, with the same key value, have similar values in other ...
0
votes
0answers
6 views

How to perform cropping on large file dataset using ImageGenerators?

I have a large dataset of RGB images with 220x220x3. I want to crop the region of interest from each of them. For that, I have defined my own function. Im trying to use this with DataImageGenerators ...
0
votes
1answer
192 views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
0
votes
2answers
23 views

Multiple Features with the same categorical value

In working with a telecommunications data set with multiple categorical variables which all depend on Internet Service, a separate categorical variable, I ran into the problem where 'No Internet ...
1
vote
0answers
24 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
0
votes
1answer
35 views

Logistics Demand Forecasting with 20k Different Time Series

I'm trying to tackle a very challenging problem and I would appreciate your help. My organization has a lot of different items which can be demanded by our clients. Those items can also be returned ...
1
vote
2answers
21 views

How to properly use oversampling without inflating results?

I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i ...
0
votes
1answer
18 views

Extracting and tagging/labeling text from pdfs [closed]

I'm trying to extract paragraphs from 230 pdfs that have similar formats, and then label the paragraphs based on the pdf's section headings. Normally, I'd convert the pdf to txt files or extract the ...
1
vote
0answers
18 views

Pre-processing packet data [closed]

I am confused about how to pre-processed packet data. Does anybody have an idea of how to encode network ports and IP addresses? My goal is to model the data in a such way that I will get a good ...
0
votes
1answer
11 views

Find differences of two protein sequence files

I have two separate files A and B of more than 100000 records of protein sequences. Now I need to find the sequences that are in ...
1
vote
0answers
17 views

What would be the best way to combine several denoised images and achieve superior denoising results? [closed]

I have generated denoised images using several models and would like to create an ensemble of these individual images to achieve superior denoising results. What would be the best way to combine (...
2
votes
1answer
31 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
0
votes
0answers
12 views

The tag preprocessing says it is a data-mining technique ! Is it a correct definition?

This site appears to give Wrong definition of preprocessing of data.Data-mining has no connection with preprocessing of data. The data preprocessing focuses on data preparation for developing software ...
1
vote
0answers
74 views

Sliding Window Normalization

I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize ...
0
votes
0answers
16 views

Pre-processing data - Dataframe manipulation Time Series

I have a question in which I'm not entirely sure in which path to take. I'd appreciate if you could point me in the right direction. Below a screenshot of the a few records of my dataset. As you can ...
0
votes
0answers
31 views

Take several points for each period in a machine learning model

Problem presentation I am working on a prediction model where I must find out if a boat will go back to the same offshore workplace after spending time in a port. When a boat is in a port, it can stay ...
1
vote
2answers
45 views

How to preprocess tensorflow imdb_review dataset [closed]

I am using tensorflow imdb_review dataset, and I want to preprocess it using Tokenizer and pad_sequences When I am using the Tokenizer instance and using the following code: ...
0
votes
0answers
27 views

How do I identify the postulated 600 object classes in OpenImages v. 6?

I have downloaded the OpenImages version 6 image sets (train, validation, test) from CVD Foundation as per instructions from the OpenImages homepage. The OpenImages homepage states that there are 600 ...
0
votes
0answers
99 views

Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

I want to train a Sequential Neural Net (NN) with Tensorflow. ...
0
votes
0answers
12 views

Handling too much range in texts' length

I have a classification problem where inputs are news article and target are topics. However, articles length are very varying from 100 words to 800! I'm using TfIdf and I think that it can puzzle the ...
1
vote
0answers
16 views

How to preprocess data for multiple rows instance? [closed]

I have a machine learning problem where I must detect anomalies in (let say) taxes declaration. So I have multiple rows for each enterprise, each row describing the tax declaration a the time (date,...
0
votes
2answers
32 views

Is there any different between feature selection and pca? If there is could anyone please kindly explain for me please?

First of sorry for asking a possibly beginner question, but i don't understand pca seems to be the same as feature selection, but when i search online they seems to be talked differently. What people ...
0
votes
1answer
1k views

How to use inverse_transform in MinMaxScaler for pred answer in a matrix

I am working on a data, for preding output, I used SVR by bellow code: ...
0
votes
0answers
29 views

How to make a Label Encoder trained on a training dataset transform an unseen value of a test dataset?

During the data preprocessing stage, I decided to apply the Label Encoding on one of the columns because it contained data points in string format. Suppose the column contains the following distinct ...
0
votes
0answers
29 views

How can I compare imputation techniques on a dataset with sci-kit learn?

I have a dataset data that has missing values. I am trying two ways of imputing these values, but I would like to compare them. In the first method I am using a ...
0
votes
0answers
13 views

Transforming binary data for decision trees

I have binary columns in my dataset (20) e.g. hot_weather, discount (y or no), where in each case 1 = yes no = 0. I am using this data on tree based methods. It is a regression problem and my RMSE is ...
3
votes
2answers
129 views

Preprocessing: StandardScaler() Do we really need mean to be zero?

For instance, many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the l1 and l2 regularizers of linear models) assume that all ...
0
votes
1answer
39 views

Normalization for a 2d input array

I am new to machine learning and trying to apply it to my problem. I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there ...
1
vote
0answers
28 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
0
votes
0answers
7 views

Factors in choosing a continuous encoder?

Apart from heuristically treating an encoder like a hyperparameter, how can one decide which encoder to use for continuous features? Rule out encoders that do/ don't accept negative values. Can we ...
2
votes
1answer
257 views

What is the best way to deal with image sizes in object detection using YOLO v. 4?

I am using YOLO v. 4 to perform object detection and classification on a large number of images. I do this through the Python interface. My images are all sorts of sizes and aspect ratios. Examples of ...
1
vote
1answer
57 views

Remove correlated features before or after splitting test and training set?

I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it ...
1
vote
1answer
46 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...

1
2 3 4 5
8