Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [preprocessing]

The tag has no usage guidance.

0
votes
0answers
16 views

How to pre-process large data for company prediction?

I have a dataset which is for 20 companies. Each company has 15 branches in 15 cities. A company has data for each branch. Each branch data is like below: ...
0
votes
1answer
23 views

What is the the cost of combining categorical variables?

I have 2 categorical variables e.g. state and city. Missing are only in city. As opposed to throwing out all observations with missing values for city or throwing out city all together I was ...
0
votes
1answer
9 views

Does discretization of continuous features also lose information about distance?

During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way. But doesn't it also lose information about distances of features? For example, if we ...
0
votes
0answers
21 views

Address Class-Imbalance Dataset using Sampling Techniques on Train, Test Dataset or Both?

I am dealing with an unbalanced dataset and I'm really confused if I should apply sampling techniques, like downsampling, smote, upsampling etc., on the train, test dataset or both? The minority ...
0
votes
2answers
23 views

Transformation of categorical variables (binary vs numerical)

When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have a feature vector with values A, B and c. The ...
0
votes
1answer
20 views

Is it correct to use non-target values of test set to engineer new features for train set?

Suppose, I have a dataset with a feature_1 value and a target value. Now, I want to engineer a new feature by creating relative value by subtracting mean from each value. Question: Can I (1) use ...
0
votes
1answer
21 views

LSTM - How to prepare train from a dataset which contains multiple observations for different events

I m using LSTM in a project related to MobiFall dataset which contains falls and daily activitives - such as walking, sitting etc - sensed by accelerometer, gyroscope and orientation sensors in x,y,z ...
1
vote
0answers
64 views

How to deal with attributes that can vary arbitrarily for each sample?

Let's say we are trying to classify cars into five different categories. For this, we have a lot of samples described by color, brand, model, year of manufacture and so on. For instance, imagine ...
0
votes
0answers
14 views

How to deal with ordered set of values with different ranges

I have data to train a model on, a mix of continuous and categorical values, one column is labeled with values like "[0,10], [10,100], [100,500], [500,]". those are ...
1
vote
0answers
14 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
0
votes
1answer
46 views

pandas: How to impute the categorical column by the nearest neighbors?

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...
0
votes
2answers
41 views

Is there any tool for data visualization and manipulation?

I have a time series data set that I need to manually label them for supervised learning. What I am doing now is using excel to plot, and when I see the pattern that I want, I hover over the data on ...
0
votes
1answer
33 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
2
votes
1answer
94 views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
1
vote
1answer
35 views

Can preprocessing the whole population cause data leakage?

Introduction I understand the problem of data leakage that could be caused by the preprocessing step when our training and test sets are just samples of an unknown population. The preprocessing ...
1
vote
0answers
26 views

Any tool that can help on manually label a time series data please?

i am working with a 10 years weekly financial time series data set, which has the standard format date open high low close volume. i would like to manually label (classify) the data set with label/...
2
votes
1answer
18 views

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
1
vote
0answers
17 views

How standardizing and/or log transformation affect prediction result in machine learning models

I recently ran an elastic net model on my data. My predictors are mostly skewed. I found my model perform slightly better when I standardize on log-transformed data than standardizing on original data....
0
votes
0answers
25 views

Discard overlapping data sets in classifications

I think I have overlapping in the training data sets. Because if I use the training set that of one label as the test documents, it will output other labels as classifications. I'm thinking of ...
1
vote
0answers
16 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
1
vote
0answers
27 views

Transposing a dataframe (time-series)

I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting): ...
0
votes
0answers
29 views

ZCA preprocessing to CIFAR-10 dataset

I am trying to preprocess my CIFAR-10 dataset using ZCA technique. The code is give below ...
5
votes
1answer
187 views

StandardScaler before and after splitting data

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
1
vote
0answers
14 views

Normalize data with uneven groups?

I have a dataset with 3 independent variables [city, industry, amount] and wish to normalize the amount. But I wish to do it with respect to industry and city. Simply grouping by the city and industry ...
1
vote
0answers
13 views

Are there some resources for filters specifically applicable in big data applications? [closed]

Are there some resources for filters specifically applicable in big data applications? Particularly, are there major differences between filter design for other domains and filter design for data ...
2
votes
3answers
35 views

When should ordinal data be represented catigorically and when as integer?

I am doing the Kaggle competition House Prices: Advanced Regression Techniques to learn more about data analysis. I would like to apply multiple models to the data(Regularized LR, Random Forests, ...
1
vote
0answers
22 views

Input data for this dataset to be feed into keras for training

Suppose I have 3 csv files which forms the dataset for training a machine learning model in Keras. file1.csv ...
1
vote
2answers
42 views

Preprocessing of Sudoku Dataset from Kaggle

Dataset: https://www.kaggle.com/bryanpark/sudoku I would like to create a neural network for this Dataset. Feature: ...
0
votes
2answers
30 views

Basics: What is the correct sequence for preparing simple data for ML?

I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one). I was just wondering what would be the best way to organise the data to avoid redundancy with the ...
0
votes
0answers
11 views

Does it make sense to preprocess (normalise or standardise) this data for GAN?

I'm working on a project where I have a dataset for a dynamical system (pendulum) containing a trajectory, energy cost and corresponding control actions (See below). I'm using a generative adversarial ...
0
votes
1answer
68 views

How to interpret Hashingvectorizer representation?

I cannot really understand the logic behind Hashingvectorizer for text feature extraction. I can follow the logic of Bag of Word or TFiDF where the features are values for all/certain words/N-grams ...
0
votes
0answers
11 views

Preprocessing during training and scoring of LSH

I am using LSH algorithm given in Spark. The dataset I am using has both numeric and categorical columns. Hence before training ...
3
votes
0answers
30 views

Feature importance over a subset of instance space instead of an entire instance space

I'm really curious if anyone has faced this problem before, or is it even widely studied at all. Imagine we have a feature that isn't important (based on many widely available and textbook feature ...
0
votes
2answers
36 views

What to do if my target variable is column of lists? [closed]

How I can transform my target variable(Y)? As it is list, I cann`t use it for fitting model, because I must use integers for fitting.
0
votes
0answers
27 views

Encrypt a data of single column of csv file in Nifi

I want to encrypt one column of my file which is present in local.What all processors will be required to fetch one column and also for encryption. I am new to Nifi. So, Any leads would be ...
0
votes
2answers
110 views

How to detect phrases from an English sentence.

The question is not about detecting keyphrases. It is about detecting a combination of words makes a valid phrase or not. For example, "John reads New York Times in New York." Here, the phrases ...
3
votes
1answer
77 views

Normalizing test data

I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the ...
0
votes
0answers
17 views

Remedy feature distribution mismatch with pre-processing

I'm quite new to machine learning and I'm working on a conditional VAE. I'm experiencing a problem where the distribution for one of the generated variables does not match the true distribution at all....
0
votes
0answers
24 views

Different approaches of creating the test set

I came across different approaches to creating a test set. Theoretically, it's quite simple, just pick some instances randomly, typically 20% of the dataset and set them aside. Below are the ...
1
vote
1answer
37 views

Dropping less frequently used categorical data?

I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite ...
2
votes
1answer
134 views

Standard correlation coefficient of various datasets

I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure 1. Why is it so ?. Even the shapes differ for ...
1
vote
3answers
47 views

Need help in understanding a small example

Pardon me, I agree the title of the question is not clear. I would like to know the understanding of below steps which are picked from the textbook "Hands on machine learning". ...
0
votes
1answer
19 views

Computer Vision: Handling dataset(3D data or scan) with different timesteps

I'm planning on training a CNN on CT scans for classification. The problem is CT scans are taken slice by slice, and in a typical scan, there could be more than 200 slices. The number of slices in a ...
1
vote
1answer
31 views

Why is it important to have sufficient number of instances in your dataset for each stratum?

As per the figure 1, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \$60,000. I didn't understand the explanation behind why housing['...
1
vote
1answer
24 views

How to check if audio samples have only noise or are silent?

I have a wav file I want to split into frames in order to feed it into a machine learning model. The problem is that the audio has silence with some noise at some points. My problem is that I do not ...
3
votes
2answers
35 views

Normalising data with multiple methods

When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score ...
0
votes
1answer
347 views

Extract all the data of a particular month from dataset of different years

In a dataset containing temperatures of different years and I want to extract data of particular month from all different years in single liner code what is the syntax? To extract all the data from a ...
1
vote
2answers
164 views

How to scale prediction back after preprocessing

So I'm a newbie to machine learning and am currently using the iris data set. I ran through a quick online tutorial about predicting stock prices and thought I'd try and do the iris one myself. The ...
1
vote
1answer
111 views

normalization/denormalization for linear regression problem

My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after ...
1
vote
0answers
76 views

Delete strings with a specific last character in tibco spotfire [closed]

I am using tibco spotfire and I have columns with data like this "0001V", "0002V", "0003N". I know data with the 'N' at the end is not valid so I need to delete the numbers which have that. I think I ...