Questions tagged [preprocessing]

The tag has no usage guidance.

78 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
15 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
2
votes
0answers
21 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
2
votes
0answers
102 views

Audio files and their corresponding spectrograms for image classification process

Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding ...
2
votes
2answers
2k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
2
votes
2answers
36 views

How to Normalise features for small datasets?

I am working with a small dataset ( N = 50 ). I would like to normalise my input features. I am facing the following issues: Because of the small size of the dataset the range of training input ...
2
votes
1answer
65 views

Clustering time series based on monotonic similarity

Context I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, ...
2
votes
2answers
126 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
2
votes
0answers
508 views

Columntransformer multiple columns with vector inputs

This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this. My question is related to the sklearn's <...
2
votes
1answer
316 views

normalization/denormalization for linear regression problem

My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after ...
2
votes
0answers
289 views

User activity representation for Prediction/ML

I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data. The raw data is a sequence of all activities by all users: ...
1
vote
0answers
35 views

Sparsify meaningful data following a Gaussian distribution

Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the meaningful information, that is the information that lies far away from the mean. One way to do ...
1
vote
0answers
7 views

fit transform with mysterious special chracters

I tried to make the bag of words ...
1
vote
2answers
33 views

How can I use mean normalization. Should I use it for numerical columns or categorical columns as well?

Should we normalize the categorical columns in our dataset? Or just the numerical columns?
1
vote
0answers
10 views

Preparing new samples for ML classifier with the same encoders

I was wondering what's the best way to preprocess new samples for my ML classifier. I have a raw data with about 3000 samples. I'm preprocessing it with some LabelEncoders and TargetEncdoders for ...
1
vote
0answers
21 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
1
vote
1answer
21 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
17 views

Labeling audio dataset

I would like to try and create my own audio dataset which I can then use to train machine learning models for classification. The data that I've gathered consists of multiple long audio files of ...
1
vote
0answers
43 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
1
vote
0answers
149 views

One hot encoding as input to recurrent neural networks

I'm trying to predict next label in a pattern based on previous labels using recurrent neural network. In total I have 100 labels Example of input pattern: ...
1
vote
0answers
20 views

How can I transform (pre-process) pure count data for PCA analysis?

If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?
1
vote
0answers
15 views

Group data without losing information

Context Imagine that I have a dataset about sending messages. Each row as user_id, a batch_id, a ...
1
vote
0answers
45 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
1
vote
0answers
121 views

Decision Tree - Preprocessing for very sparse features

How do we pre-process data for very sparse features for a decision tree? From this Turi documentation for decision trees It mentioned this: Why chose decision trees? Different kinds of models ...
1
vote
0answers
59 views

Image preprocessing: How to resize / align / cut images of various sizes?

I want to create a new dataset for image recognition. If I have Object A that I want to recognize in images, and multiple images of various different sizes, how do I preprocess them so that they ...
1
vote
0answers
112 views

Should I remove the trend from timeseries when using DeepAR

I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM) I figured out from the paper that SageMaker's ...
1
vote
0answers
31 views

Construction Adjacency matrix for GNN

A bit of a novice question, my data comes in the form of 2D hit points, my goal is to perform node/edge classification to find out whether this graph is my signal or background. My question is, when ...
1
vote
2answers
51 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
1
vote
0answers
20 views

Specific data formatting techniques for discontiguous time series?

I'm facing a predicting problem for food alerts. The goal is to predict the variables of the most probable alert in the next x days (also any information I could get about future alerts is really ...
1
vote
0answers
85 views

Remove attributes with missing values exceeding a given threshold in WEKA

I imported csv file into WEKA, i have features that have missing value that has missing value percentage of 70% or above, i want to remove these features by weka or also sort that features by missing ...
1
vote
0answers
69 views

How to deal with attributes that can vary arbitrarily for each sample?

Let's say we are trying to classify cars into five different categories. For this, we have a lot of samples described by color, brand, model, year of manufacture and so on. For instance, imagine ...
1
vote
0answers
134 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
1
vote
2answers
63 views

Is there any tool for data visualization and manipulation?

I have a time series data set that I need to manually label them for supervised learning. What I am doing now is using excel to plot, and when I see the pattern that I want, I hover over the data on ...
1
vote
0answers
44 views

Any tool that can help on manually label a time series data please?

i am working with a 10 years weekly financial time series data set, which has the standard format date open high low close volume. i would like to manually label (classify) the data set with label/...
1
vote
0answers
54 views

How standardizing and/or log transformation affect prediction result in machine learning models

I recently ran an elastic net model on my data. My predictors are mostly skewed. I found my model perform slightly better when I standardize on log-transformed data than standardizing on original data....
1
vote
0answers
133 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
1
vote
0answers
60 views

Transposing a dataframe (time-series)

I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting): ...
1
vote
0answers
33 views

Normalize data with uneven groups?

I have a dataset with 3 independent variables [city, industry, amount] and wish to normalize the amount. But I wish to do it with respect to industry and city. Simply grouping by the city and industry ...
1
vote
0answers
79 views

Input data for this dataset to be feed into keras for training

Suppose I have 3 csv files which forms the dataset for training a machine learning model in Keras. file1.csv ...
1
vote
0answers
21 views

Variable TimeStep in Spatio Temporal 3DConvNet

I'm working on a new project on climate data. I suppose my output $Y(H*W*1)$ is a consequence of a given data scenario $X(C=4*T*H*W)$ and an initial state $Y0(H*W*1)$. I've chosen a temporal ...
1
vote
0answers
23 views

Are annotated audio datasets augmented with mutated versions the way image datasets are?

Data augmentation is very standard for annotated image datasets for tasks like image labelling. Images are flipped, rotated, pixelated and so on, to add more training data and make the system robust ...
1
vote
0answers
110 views

Chunker/shallow parser for spoken language

I'm trying to extract NPs from transcribed spoken text, such as um it's the bl- it's the blue one in the right no left hand corner which contains e.g. fillers (e.g. um) and disfluencies (e.g. bl-,...
1
vote
0answers
153 views

how is countvectorizer used in real production environment?

how is countvectorizer used in real production environment? do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?...
1
vote
0answers
86 views

Transformation of Dependent and Independent Variables

I have a few Independent variables that's normal and a Dependent variables that's skewed , I pick log(feature+SHIFT) to correct skewness. The procedure I follow to get prediction is just take exp(...
1
vote
0answers
438 views

Clustering bitcoin addresses with k-means - how would one prepare input

I am looking to perform clustering on bitcoin addresses within the blockchain. I have generated a graph structure of the blockchain with a source address, destination address, value of transaction, in-...
1
vote
0answers
50 views

How do I convert a series of timestamps in seconds to milliseconds in order to distribute them smoothly

I have a script that collects memory usage data from a device running a Linux stack. I collect samples roughly every 200ms and write the measurement to a csv file with a timestamp. The problem is ...
1
vote
0answers
297 views

How to best represent rate or proportion as a feature?

Given two features in a data set: NFObs (Number of failed observations) and NObs (Number of total observations), I'd like to ...
1
vote
1answer
375 views

What metrics must i use in my data(unstructured) preprocessing research?

i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats). i wish to prove preprocessing improves the content quality. are there metrics available to prove ...
0
votes
1answer
5 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
0
votes
0answers
8 views

prediction with un-obligatory features

I am pre-processing a real-world dataset with some features with missing values. these features are not mandatory - the user doesn't have to provide them. This means the values are not missing at ...
0
votes
0answers
10 views

Preprocessing data in image segmentation problem

I am implementing a research paper on image segmentation. Following are the image segmentation steps which are to be done before training its network- 1.Following image normalization is used- ...