Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Selecting audio pre-processing parameters for ASR

As far as I understand, audio in many speech recognition systems goes through the following pre-processing steps: Slice audio stream into frames of size frame_size ...
4
votes
1answer
25 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
3
votes
1answer
317 views

Handling features with multiple values per instance in Python for Machine Learning model

I have a dataset which contains medical data about children and I am developing a predictive machine learning model to predict adverse pregnancy outcomes. The dataset contains mostly features with a ...
1
vote
1answer
62 views

How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument?

The only argument inside the fit function of SimpleImputer is: 'Age'. Yet the returned output worked on the 'Salary' column as well. That is what I am unable to understand. Here is my code (...
2
votes
3answers
2k views

Count the max number of consecutive 1 and 0 in Pandas Dataframe [closed]

Hey I have the following Dataset import pandas as pd df = pd.DataFrame({ 'column1': [0,0,1,0,1,0,0,1,1,0,1,1,1]}) I want to be able to count the number of ...
3
votes
1answer
75 views

What is the proper order of normalization steps before and after splitting data

I use a classification model on time-series data where I normalize the data before splitting the data into train and test. Now, I know that train and test data should be treated separately to prevent ...
16
votes
3answers
31k views

Image resizing and padding for CNN

I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized ...
1
vote
1answer
29 views

How do I deal with data that has only limited target values?

I'm currently working on a small project using the D1NAMO dataset (1). I want to predict the glucose level (that is given in the dataset) based on several features: accelerometer data, heartbeat (ECG) ...
1
vote
2answers
296 views

What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what ...
1
vote
2answers
56 views

Machine Learning - Input Prepocessing - NLP email classification model

So I created a model which classifies emails into different categories, just like a spam filter. I deployed the model as a webservice, no problem with that but I can’t get my head around how I would ...
1
vote
0answers
95 views

Split-Half Reliability Python [closed]

I have a pandas dataframe of baseball stats that I have transposed that looks as follows: Each column represents a separate plate appearance for a player. Each player can (and usually is) represented ...
0
votes
1answer
25 views

How to adjust/smooth a certain number using constants or rules

Hi, I am handling a dataset with a customer purchase history. The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage. There are two ...
4
votes
2answers
1k views

Is it acceptable not to transform() test data after train data is being fit_transform()-ed

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
0
votes
1answer
95 views

How to make prediction using tensorflow models?

As a newbie to tensorflow, I am using this tutorial from google for binary classification using a simple dense neural network. The slightly annoying thing about this (and a few other) tutorials is ...
0
votes
1answer
899 views

Data augmentation for multiple output heads in Keras

I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a ...
1
vote
1answer
34 views

Web Scraping: Multiple small files or one large file?

I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each ...
6
votes
2answers
272 views

How distribution of data effects model performance?

I am working on House Prices: Advanced Regression Techniques dataset. I was going through some kernels noticed many people converted SalePrice to ...
1
vote
1answer
25 views
1
vote
0answers
75 views

How to handle negative delays when predicting flight delays?

I am working with the nycflights dataset. My goal is to predict departure and arrival time delays using random forests. My ...
0
votes
1answer
402 views

What is the efficient way to use apply method in column of pandas Dataframe for large dataset?

I have a dataset of approximate 1 hundred thousand records. I want to use apply method in each of the records for further data processing but it takes very long time to process (As apply method works ...
0
votes
1answer
27 views

How to seperate text lines from txt files in python?

I am playing with a .txt file in python for EDA and I want to seperate the lines from this fashion: ...
0
votes
1answer
31 views

What pre-processing of the image is needed before feeding it into the convolutional neural network?

I can't figure out what preprocessing of the image is needed before feeding it into the convolutional neural network. For example, I want to recognize circles on a 1000 by 1000 px photo. The learning ...
0
votes
2answers
29 views

One attribute includes another attribute

I have a telecom dataset that has many attributes, among these attributes, there is "Voice mail plan" attribute that takes yes or no, and another attribute is "voice mail calls" which has many values, ...
0
votes
2answers
7k views

Extract all the data of a particular month from dataset of different years

In a dataset containing temperatures of different years and I want to extract data of particular month from all different years in single liner code what is the syntax? To extract all the data from a ...
0
votes
1answer
149 views

Pre-processing - Removing outliers

I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data: It looks like outliers are 4,5,6 and should be removed. Now ...
0
votes
0answers
89 views

How do I extract album and song titles from this plain text file?

Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the ...
3
votes
1answer
352 views

Why is oversampling outperforming class weight?

I have a dataset that is highly imbalanced. One class has 412 (class 0) samples while the other has 67215 (class 1) samples. For its classification, I am using MLP. When I use class weight of 165 for ...
0
votes
0answers
30 views

Alternatives to reshaping the data

I have a medical dataset which looks like this: ...
2
votes
0answers
57 views

Forecast multiple unevenly spaced time series

I am building a time-series forecasting model to predict some patterns in climatological data. The dataset consists of many (2 mln) time series which look for example as: However the observations ...
7
votes
1answer
7k views

How to implement global contrast normalization in python?

I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast ...
3
votes
1answer
136 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
1
vote
0answers
15 views

Is there any problem using Euclidean Distance for a data set that includes a lot of values between 0 and 1?

I have a data set which includes 3 types of financial data (P/E, ROA, EPS growth) for each stock in a list of stocks. My goal is to use K-Medoids to cluster these stocks into groups based on this data....
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
3
votes
1answer
4k views

Iterate over multiple dataframe rows at the same time

I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values. I need to loop over ...
1
vote
0answers
25 views

svd to reduce text matricies

I am having problems with very big texts. after preprocessing each of my docs represented as a matrix of sentences, where each sentence represented as encoded words (each word have a unique vocab ...
2
votes
3answers
1k views

Transformation of categorical variables (binary vs numerical)

When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have a feature vector with values A, B and c. The ...
1
vote
2answers
151 views

Pre-processing mixed data prior to clustering

I am new to hierarchical clustering, and wish to perform clustering on mixed data. I am slightly confused on the necessary pre-processing steps. I understand how to pre-process purely continuous data, ...
1
vote
1answer
42 views

Representing 2 dimensional data for facial recognition

My goal is to train a neural network to recognize faces based on a list of landmark points generated by Google Firebase ML-Kit. Since I just started familiarizing myself with ML, I only want my model ...
0
votes
2answers
222 views

Preprocessing of 3D CAD files for Keras Conv3D input

I'd like to apply some machine learning on 3D CAD data. File format should ideally be mesh-based like STL. Keras offers 3D convolutional layers (https://keras.io/layers/convolutional/), so it can ...
0
votes
1answer
54 views

Should we first remove the unwanted independent variable or split the data into test train data?

I wanted to know if we are dropping any independent variable as it has too many missing values(~75% or more) from the data then should we do it before splitting the data or after splitting the data
0
votes
1answer
229 views

I have a pandas dataframe and i need to clear all the special characters other than space

Input: import pandas as pd df=pd.read_excel("OCRFinal.xlsx") df['OCR_Text']=df['OCR_Text'].str.replace(r'\W+'," ") print(df['OCR_Text']) Output: The excel ...
1
vote
0answers
84 views

MinMaxScaling vs L1/L2-Normalization

I'm wondering about the difference or the application of the different types of rescaling data. So far, I'm aware that standardization assumes the data has a gaussian distribution. So if this is the ...
1
vote
1answer
2k views

How to preprocess data for Word2Vec?

I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?
2
votes
1answer
51 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm. I made sure that the class counts are balanced. i.e., an equal number of ...
3
votes
2answers
86 views

How can I perform categorical encoding when the dataset is too large for memory?

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
5
votes
2answers
2k views

Columntransformer multiple columns with vector inputs

This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this. My question is related to the sklearn's <...
2
votes
1answer
406 views

What metrics must i use in my data(unstructured) preprocessing research?

i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats). i wish to prove preprocessing improves the content quality. are there metrics available to prove ...
2
votes
1answer
96 views

Word Embedding or Hash?

In my dataset I have a 'text' column and a 'followers' column containing lists of follower IDs, i.e. '1093777852477116417, 936194589043683328,...'. Some of the 'followers' values contain thousands of ...
2
votes
0answers
178 views

Will flattening multivariate time series data before clustering make the results meaningless?

I have a large number of financial time series that I wish to do cluster analysis on. Each time series has the same length and spans multiple years of daily data (returns, volatility, etc.). As part ...
1
vote
0answers
55 views

Can anyone direct me on how to got to [closed]

I am a newbie to python. I have uploaded about 2000 images into my google drive, assigned a variable to all content in the folder and tried to split the folder's content into training and testing, ...

1 2 3
4
5
8