Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [preprocessing]

The tag has no usage guidance.

1
vote
0answers
14 views

One hot encoding as input to recurrent neural networks

I'm trying to predict next label in a pattern based on previous labels using recurrent neural network. In total I have 100 labels Example of input pattern: ...
0
votes
0answers
10 views

Image Labeling Tool

Problem Now I am working on a project to identify the subjects in the image is man, woman or ...
0
votes
1answer
15 views

Suggestion on Preprocessing dataset

I am trying to preprocess my dataset and needs some suggestion on it. The training data shape is : (166573, 14) The distribution of features : As you can see, only the first 4 columns go to ...
0
votes
0answers
7 views

Using Keras.utils.Sequence on out of memory dataset

I'm trying to implement a sub-class of keras.util.sequence so that i can load data faster into the function fit_generator. Currently I have a multi-variate time series (multiple features) file which ...
0
votes
0answers
15 views

How can I transform (pre-process) pure count data for PCA analysis?

If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?
0
votes
1answer
15 views

One attribute includes another attribute

I have a telecom dataset that has many attributes, among these attributes, there is "Voice mail plan" attribute that takes yes or no, and another attribute is "voice mail calls" which has many values, ...
0
votes
1answer
41 views

Training a LSTM on a time serie containing multiple inputs for each timestep

I am trying to train a LSTM in order to use it for forecasting : the problem is basically a multivariate multi-steps time series problem. It is simply an experiment to see how statistical models (...
0
votes
0answers
18 views

How can we add preprocessing steps, in the keras sequential model itself?

Is there a way to add a layer which includes my preprocessing steps in this sequential model.For example ...
2
votes
1answer
42 views

SVM SMOTE fit_resample() function runs forever with no result

Problem fit_resample(X,y) is taking too long to complete execution for 2million rows. Dataset specifications I have a labeled dataset about network features, ...
2
votes
1answer
33 views

Preprocess image data to classify objects based on shape

Currently I'm trying to build a neural network that is able to classify different types of bottles on an image solely based on the shape. The bottles have no label and at first I only used beer and ...
0
votes
1answer
25 views

Is there any NLP library or package which can help in adding coma, punctuations, new line appropriately in text?

I have movie transcript, where no coma, punctuations or new line. Is there any NLP technique which can help to implement this?
2
votes
1answer
28 views

Python - Create many dummy variables from one text variable?

I'm trying to create dummy variables for a variable that has text data in rows. Data in 1st row is: ...
2
votes
0answers
34 views

Audio files and their corresponding spectrograms for image classification process

Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding ...
1
vote
0answers
14 views

Group data without losing information

Context Imagine that I have a dataset about sending messages. Each row as user_id, a batch_id, a ...
0
votes
1answer
30 views

Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables. To create dummy variables I used pd.get_dummies() before the split. I now ...
0
votes
1answer
54 views

what is correct way to perform normalization on data in Auto encoder?

working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of ...
0
votes
0answers
16 views

Google Data to estimate popularity?

I have Google point of interest data including each poi‘s rating and number of rates. Do you think I can use this somehow to estimate popularity of certain regions? I am not sure if this is really ...
1
vote
0answers
29 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
1
vote
0answers
30 views

Decision Tree - Preprocessing for very sparse features

How do we pre-process data for very sparse features for a decision tree? From this Turi documentation for decision trees It mentioned this: Why chose decision trees? Different kinds of models ...
0
votes
1answer
26 views

How to deal with name strings in large data sets for ML?

My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later. Some word embedding techniques were used for longer text ...
2
votes
1answer
102 views

Why does not log transformation make the data normalized?

Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following ...
1
vote
1answer
40 views

Mutate with dynamic column names dplyr

Hi I have this dataset (It has many more columns) ...
0
votes
1answer
22 views

How to discretize certain features with a feature set?

I am working with typing data with timing features(unit: ms) and some of the features are based on the keyboard keyCodes(positive integers, range:[8, 222]). Currently, I use ...
1
vote
0answers
36 views

Image preprocessing: How to resize / align / cut images of various sizes?

I want to create a new dataset for image recognition. If I have Object A that I want to recognize in images, and multiple images of various different sizes, how do I preprocess them so that they ...
1
vote
0answers
36 views

Should I remove the trend from timeseries when using DeepAR

I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM) I figured out from the paper that SageMaker's ...
1
vote
2answers
144 views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
2
votes
2answers
29 views

How to Normalise features for small datasets?

I am working with a small dataset ( N = 50 ). I would like to normalise my input features. I am facing the following issues: Because of the small size of the dataset the range of training input ...
2
votes
1answer
29 views

Clustering time series based on monotonic similarity

Context I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, ...
1
vote
0answers
16 views

Construction Adjacency matrix for GNN

A bit of a novice question, my data comes in the form of 2D hit points, my goal is to perform node/edge classification to find out whether this graph is my signal or background. My question is, when ...
2
votes
1answer
61 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
1
vote
0answers
21 views

Calculating unaffected data [closed]

Some data sets values are missing. Missing values are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would remain unaffected in %. Why?...
0
votes
0answers
19 views

Checking skewness of variables

I follow a tutorial for the current dataset I'm working on. In this tutorial the target variable is check for skewness, and corrected het for by by taking the log transformation. Hereafter, skewness ...
1
vote
2answers
37 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
0
votes
1answer
47 views

LSTM for prediction of next location step - help with standardization

I have a few questions regarding the topic and I hope someone might have experience with any of them. What I am trying to do is train an LSTM network, whose input is a sequence of N steps in a XYZ ...
1
vote
0answers
14 views

Specific data formatting techniques for discontiguous time series?

I'm facing a predicting problem for food alerts. The goal is to predict the variables of the most probable alert in the next x days (also any information I could get about future alerts is really ...
-1
votes
1answer
24 views

Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query). What are the pros and cons of: 1) Running the ...
1
vote
1answer
54 views

Encoding multiple observations from the same feature space

My data contains multiple observations of categorical feature.The feature space is medical symptoms, so the data for this feature is like : ['fever','pain','yellow skin' .... ] .The amount of symptoms ...
2
votes
1answer
16 views

Image normalisation methods

I have found some research papers specifying explicitly the normalisation technique they used to get the results. What difference do IMG /255.0 And ...
1
vote
0answers
48 views

Remove attributes with missing values exceeding a given threshold in WEKA

I imported csv file into WEKA, i have features that have missing value that has missing value percentage of 70% or above, i want to remove these features by weka or also sort that features by missing ...
1
vote
1answer
57 views

Cleaning data automatically [closed]

Do you use automatic cleaning tools for data? I mean something similar to h2o.ai's auto ml function but applied to preprocessing data. Or do you always clean data 'by hand'.
1
vote
2answers
52 views

data pre-processing before image classification

I'm working on a machine learning project, Images classification (shape: 100 x 100)-> (vector of 10000), I did some pre-processing before applying decision trees algorithm , I got an accuracy of 55 % ...
1
vote
0answers
304 views

Columntransformer multiple columns with vector inputs

This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this. My question is related to the sklearn's <...
0
votes
1answer
37 views

What is the the cost of combining categorical variables?

I have 2 categorical variables e.g. state and city. Missing are only in city. As opposed to throwing out all observations with missing values for city or throwing out city all together I was ...
0
votes
1answer
30 views

Does discretization of continuous features also lose information about distance?

During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way. But doesn't it also lose information about distances of features? For example, if we ...
0
votes
2answers
53 views

Transformation of categorical variables (binary vs numerical)

When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have a feature vector with values A, B and c. The ...
0
votes
1answer
31 views

Is it correct to use non-target values of test set to engineer new features for train set?

Suppose, I have a dataset with a feature_1 value and a target value. Now, I want to engineer a new feature by creating relative value by subtracting mean from each value. Question: Can I (1) use ...
0
votes
1answer
111 views

LSTM - How to prepare train from a dataset which contains multiple observations for different events

I m using LSTM in a project related to MobiFall dataset which contains falls and daily activitives - such as walking, sitting etc - sensed by accelerometer, gyroscope and orientation sensors in x,y,z ...
1
vote
0answers
68 views

How to deal with attributes that can vary arbitrarily for each sample?

Let's say we are trying to classify cars into five different categories. For this, we have a lot of samples described by color, brand, model, year of manufacture and so on. For instance, imagine ...
1
vote
0answers
62 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
0
votes
1answer
271 views

pandas: How to impute the categorical column by the nearest neighbors?

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...