Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Input pipeline with an autoencoder and tf.data

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...
0
votes
1answer
10 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
0
votes
0answers
6 views

Small object detection preprocessing

I am currently working on an object detection project in which I amtrying to detect very small objects 50x50 object in a 2k image. EfficientDet produced a very low result if I just put the raw ...
0
votes
0answers
14 views

How to fill missing latitude and longitude values in time series data?

I have time-series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
1
vote
2answers
30 views

How to convert longitude and latitude in time series data from daily to weekly?

I have time series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
0
votes
0answers
27 views

Predict on new data / Model Deployment

I do not undestand, how the deployment of a ML-model works in the reality. A given dataset needs to be mostly time pre-processed (for example One Hot Encoding). After will be a model cretead and ...
1
vote
1answer
14 views
0
votes
1answer
13 views

Storing and loading bottleneck features for transfer learning on large data sets (Keras)

I would like to apply transfer learning on a pretty large image data set in order to solve a classification problem. Currently I load a pre-trained net without the top layers, add my own top layers, ...
0
votes
1answer
38 views

Can someone clarify what the linear assumption of PCA is?

0 For the past few hours I've been trying to search what this linear assumption is. Some of the articles states that that your independent variables have to be linear in relationship and need some ...
0
votes
1answer
10 views

Treat similar observations in a classification problem

I have a dataset of about 200k rows and I'm working on a classification problem. Grouping the dataset by a key variable, I noticed that some rows, with the same key value, have similar values in other ...
0
votes
0answers
6 views

How to perform cropping on large file dataset using ImageGenerators?

I have a large dataset of RGB images with 220x220x3. I want to crop the region of interest from each of them. For that, I have defined my own function. Im trying to use this with DataImageGenerators ...
0
votes
0answers
26 views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
0
votes
2answers
23 views

Multiple Features with the same categorical value

In working with a telecommunications data set with multiple categorical variables which all depend on Internet Service, a separate categorical variable, I ran into the problem where 'No Internet ...
1
vote
0answers
20 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
0
votes
1answer
27 views

Logistics Demand Forecasting with 20k Different Time Series

I'm trying to tackle a very challenging problem and I would appreciate your help. My organization has a lot of different items which can be demanded by our clients. Those items can also be returned ...
1
vote
2answers
18 views

How to properly use oversampling without inflating results?

I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i ...
0
votes
1answer
17 views

Extracting and tagging/labeling text from pdfs [closed]

I'm trying to extract paragraphs from 230 pdfs that have similar formats, and then label the paragraphs based on the pdf's section headings. Normally, I'd convert the pdf to txt files or extract the ...
1
vote
0answers
18 views

Pre-processing packet data [closed]

I am confused about how to pre-processed packet data. Does anybody have an idea of how to encode network ports and IP addresses? My goal is to model the data in a such way that I will get a good ...
0
votes
1answer
11 views

Find differences of two protein sequence files

I have two separate files A and B of more than 100000 records of protein sequences. Now I need to find the sequences that are in ...
1
vote
0answers
17 views

What would be the best way to combine several denoised images and achieve superior denoising results? [closed]

I have generated denoised images using several models and would like to create an ensemble of these individual images to achieve superior denoising results. What would be the best way to combine (...
2
votes
1answer
31 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
0
votes
0answers
12 views

The tag preprocessing says it is a data-mining technique ! Is it a correct definition?

This site appears to give Wrong definition of preprocessing of data.Data-mining has no connection with preprocessing of data. The data preprocessing focuses on data preparation for developing software ...
1
vote
0answers
46 views

Sliding Window Normalization

I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize ...
0
votes
0answers
16 views

Pre-processing data - Dataframe manipulation Time Series

I have a question in which I'm not entirely sure in which path to take. I'd appreciate if you could point me in the right direction. Below a screenshot of the a few records of my dataset. As you can ...
0
votes
0answers
31 views

Take several points for each period in a machine learning model

Problem presentation I am working on a prediction model where I must find out if a boat will go back to the same offshore workplace after spending time in a port. When a boat is in a port, it can stay ...
1
vote
2answers
38 views

How to preprocess tensorflow imdb_review dataset [closed]

I am using tensorflow imdb_review dataset, and I want to preprocess it using Tokenizer and pad_sequences When I am using the Tokenizer instance and using the following code: ...
0
votes
0answers
25 views

How do I identify the postulated 600 object classes in OpenImages v. 6?

I have downloaded the OpenImages version 6 image sets (train, validation, test) from CVD Foundation as per instructions from the OpenImages homepage. The OpenImages homepage states that there are 600 ...
0
votes
0answers
54 views

Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

I want to train a Sequential Neural Net (NN) with Tensorflow. ...
0
votes
0answers
12 views

Handling too much range in texts' length

I have a classification problem where inputs are news article and target are topics. However, articles length are very varying from 100 words to 800! I'm using TfIdf and I think that it can puzzle the ...
1
vote
0answers
16 views

How to preprocess data for multiple rows instance? [closed]

I have a machine learning problem where I must detect anomalies in (let say) taxes declaration. So I have multiple rows for each enterprise, each row describing the tax declaration a the time (date,...
0
votes
2answers
31 views

Is there any different between feature selection and pca? If there is could anyone please kindly explain for me please?

First of sorry for asking a possibly beginner question, but i don't understand pca seems to be the same as feature selection, but when i search online they seems to be talked differently. What people ...
0
votes
1answer
603 views

How to use inverse_transform in MinMaxScaler for pred answer in a matrix

I am working on a data, for preding output, I used SVR by bellow code: ...
0
votes
0answers
25 views

How to make a Label Encoder trained on a training dataset transform an unseen value of a test dataset?

During the data preprocessing stage, I decided to apply the Label Encoding on one of the columns because it contained data points in string format. Suppose the column contains the following distinct ...
0
votes
0answers
28 views

How can I compare imputation techniques on a dataset with sci-kit learn?

I have a dataset data that has missing values. I am trying two ways of imputing these values, but I would like to compare them. In the first method I am using a ...
0
votes
0answers
13 views

Transforming binary data for decision trees

I have binary columns in my dataset (20) e.g. hot_weather, discount (y or no), where in each case 1 = yes no = 0. I am using this data on tree based methods. It is a regression problem and my RMSE is ...
3
votes
2answers
112 views

Preprocessing: StandardScaler() Do we really need mean to be zero?

For instance, many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the l1 and l2 regularizers of linear models) assume that all ...
0
votes
1answer
33 views

Normalization for a 2d input array

I am new to machine learning and trying to apply it to my problem. I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there ...
1
vote
0answers
26 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
0
votes
0answers
7 views

Factors in choosing a continuous encoder?

Apart from heuristically treating an encoder like a hyperparameter, how can one decide which encoder to use for continuous features? Rule out encoders that do/ don't accept negative values. Can we ...
2
votes
1answer
117 views

What is the best way to deal with image sizes in object detection using YOLO v. 4?

I am using YOLO v. 4 to perform object detection and classification on a large number of images. I do this through the Python interface. My images are all sorts of sizes and aspect ratios. Examples of ...
1
vote
1answer
34 views

Remove correlated features before or after splitting test and training set?

I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it ...
1
vote
1answer
41 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...
0
votes
0answers
58 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
0
votes
2answers
191 views

Error when trying .transform for OrdinalEncoder from Scikit Learn

I'm having a lot of issues using scikit learn recently and was hoping someone could help me with my problem. I can use other methods to ordinal encode but i want to figure this one out. ...
0
votes
0answers
4 views

While Merging image datasets which of the image parameters should be prepossessed/Normalized before giving to a CNN Model?

When two datasets are merged or images of different parameters size, dimension, Format are combine which parameters of the datasets should be normalized/ pre-processed before giving it to a model?
0
votes
1answer
32 views

What will happen if almost constant values for features?

In a problem of an epidemiology dataset, is it desirable to keep the features that have almost constant values? For example, In case of the feature, type_of_residence Large for 97 percent and Small ...
0
votes
1answer
22 views

Cleaning NaNs with averages pre or post split? [duplicate]

I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode. Technically, the validation/ test data has never been seen before - so how could I include it in the ...
0
votes
1answer
120 views

Pandas replace column values by condition with averages based on a value in another column

I have a dataframe with people's CV data. Among others, there's a column with years of experience, and a column with age. Some people stated their age and experience in a way that experience > age. ...
0
votes
0answers
12 views

Bayesian Network Modeling on time series data with constant discrete features

I'm trying to model a dynamic bayesian network that can infer relationships between traffic sate of different road links over time. The main theme of mystudy is to model the relationship of change in ...
3
votes
1answer
52 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...

1
2 3 4 5
8