Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Preprocessing data in image segmentation problem

I am implementing a research paper on image segmentation. Following are the image segmentation steps which are to be done before training its network- 1.Following image normalization is used- ...
2
votes
0answers
21 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
0
votes
1answer
21 views

Positively skewed target label in regression

I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-...
1
vote
2answers
24 views

How to export PCA to use in another program

I'm trying to write a random forest classifier for a very large dataset, as such as part of the pre-processing i have applied PCA to reduce from 643 features to 5 PC's. Is it possible to export these ...
1
vote
0answers
18 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
0
votes
1answer
19 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
0
votes
0answers
7 views

How would one reduce dimensionality/covariance in a dataset with nonlinearly covariant variables? Is M-SSA a no-go?

I am familiar with the Principal Component Analysis method of covariance and dimensionality reduction. I am considering using its multivariate time series brother, Multivariate Singular Spectrum ...
0
votes
2answers
32 views

One Hot Label Encoding Scikit_learn convert back to Data Frame

I have a data frame with 4 features and 1 target. The 4 features are 3 categorical and 1 numerical. I created X which is a new data frame for the 3 categorical features. I use one hot label encoding ...
0
votes
0answers
20 views

preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
0
votes
0answers
14 views

Handling collection of featurevectors for classification

I have a data set where devices are represented by a collection of variables. These variables consist of several properties like a name, datatype, driver, limit values, etc. (mixed data; quantitative ...
0
votes
0answers
21 views

Pre-processing data to make predictions on deployed Sklearn model

I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...) I have used pickle to ...
0
votes
0answers
17 views

Feature vector of linear model

I read this paper that applies logistic regression to a dataset generated from a simulation they created. The dataset contains a set of binary vectors (called challenges) that looks like this: ...
0
votes
0answers
15 views

ZCA: Covariance of Large Image Database

I have an overall question if my method is sound, so please bear with me in this description :). I have a large image database and I wanted to create a preprocessing step using ZCA. The issue is that ...
0
votes
2answers
46 views

sklearn.preprocessing.MinMaxScaler: non-broadcastable output shape error

Why I am getting the following error: ValueError: non-broadcastable output operand with shape (1,1) doesn't match the broadcast shape (1,2) While executing: ...
0
votes
1answer
68 views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
0
votes
0answers
8 views

How to avoid the alternating conditional expectations from transforming most of the response data to very close values?

I am applying alternating conditional expectations (ACE) in a forward stepwise manner, similarly to the authors of the original ACE paper. My dataset has 103 predictor variables $x_i$, one response ...
1
vote
4answers
59 views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
1
vote
1answer
20 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
12 views

Labeling audio dataset

I would like to try and create my own audio dataset which I can then use to train machine learning models for classification. The data that I've gathered consists of multiple long audio files of ...
0
votes
1answer
31 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
0
votes
2answers
57 views

Understanding missing values in dataset

I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) ...
0
votes
0answers
5 views

Class-inflated factor. At what point do we decide to remove it?

If we have a dataset with a few string-type factors that have a lot of one class, at what point do we decide to remove the said class? This question just came to mind since I was practicing on a ...
1
vote
1answer
34 views

Scaling features separately?

I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship ...
1
vote
0answers
42 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
6
votes
2answers
72 views

Best way to scale across different datasets

I have come across a peculiar situation when preprocessing data. Let's say I have a dataset A. I split the dataset into A_train ...
1
vote
0answers
76 views

One hot encoding as input to recurrent neural networks

I'm trying to predict next label in a pattern based on previous labels using recurrent neural network. In total I have 100 labels Example of input pattern: ...
0
votes
0answers
17 views

Image Labeling Tool

Problem Now I am working on a project to identify the subjects in the image is man, woman or ...
0
votes
1answer
26 views

Suggestion on Preprocessing dataset

I am trying to preprocess my dataset and needs some suggestion on it. The training data shape is : (166573, 14) The distribution of features : As you can see, only the first 4 columns go to ...
0
votes
0answers
82 views

Using Keras.utils.Sequence on out of memory dataset

I'm trying to implement a sub-class of keras.util.sequence so that i can load data faster into the function fit_generator. Currently I have a multi-variate time series (multiple features) file which ...
1
vote
0answers
20 views

How can I transform (pre-process) pure count data for PCA analysis?

If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?
0
votes
1answer
16 views

One attribute includes another attribute

I have a telecom dataset that has many attributes, among these attributes, there is "Voice mail plan" attribute that takes yes or no, and another attribute is "voice mail calls" which has many values, ...
0
votes
1answer
121 views

Training a LSTM on a time serie containing multiple inputs for each timestep

I am trying to train a LSTM in order to use it for forecasting : the problem is basically a multivariate multi-steps time series problem. It is simply an experiment to see how statistical models (...
0
votes
0answers
34 views

How can we add preprocessing steps, in the keras sequential model itself?

Is there a way to add a layer which includes my preprocessing steps in this sequential model.For example ...
3
votes
1answer
188 views

SVM SMOTE fit_resample() function runs forever with no result

Problem fit_resample(X,y) is taking too long to complete execution for 2million rows. Dataset specifications I have a labeled dataset about network features, ...
2
votes
1answer
36 views

Preprocess image data to classify objects based on shape

Currently I'm trying to build a neural network that is able to classify different types of bottles on an image solely based on the shape. The bottles have no label and at first I only used beer and ...
0
votes
1answer
27 views

Is there any NLP library or package which can help in adding coma, punctuations, new line appropriately in text?

I have movie transcript, where no coma, punctuations or new line. Is there any NLP technique which can help to implement this?
2
votes
1answer
112 views

Python - Create many dummy variables from one text variable?

I'm trying to create dummy variables for a variable that has text data in rows. Data in 1st row is: ...
2
votes
0answers
89 views

Audio files and their corresponding spectrograms for image classification process

Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding ...
1
vote
0answers
15 views

Group data without losing information

Context Imagine that I have a dataset about sending messages. Each row as user_id, a batch_id, a ...
0
votes
1answer
242 views

Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables. To create dummy variables I used pd.get_dummies() before the split. I now ...
0
votes
1answer
67 views

what is correct way to perform normalization on data in Auto encoder?

working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of ...
1
vote
0answers
42 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
1
vote
0answers
93 views

Decision Tree - Preprocessing for very sparse features

How do we pre-process data for very sparse features for a decision tree? From this Turi documentation for decision trees It mentioned this: Why chose decision trees? Different kinds of models ...
0
votes
1answer
28 views

How to deal with name strings in large data sets for ML?

My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later. Some word embedding techniques were used for longer text ...
2
votes
1answer
269 views

Why does not log transformation make the data normalized?

Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following ...
1
vote
1answer
162 views

Mutate with dynamic column names dplyr

Hi I have this dataset (It has many more columns) ...
0
votes
1answer
32 views

How to discretize certain features with a feature set?

I am working with typing data with timing features(unit: ms) and some of the features are based on the keyboard keyCodes(positive integers, range:[8, 222]). Currently, I use ...
1
vote
0answers
54 views

Image preprocessing: How to resize / align / cut images of various sizes?

I want to create a new dataset for image recognition. If I have Object A that I want to recognize in images, and multiple images of various different sizes, how do I preprocess them so that they ...
1
vote
0answers
85 views

Should I remove the trend from timeseries when using DeepAR

I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM) I figured out from the paper that SageMaker's ...
1
vote
2answers
1k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...