Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
558 views

Converting nominal data to numeric - is using dictionaries the right approach?

Currently I have a data-set with roughly 7 types of nominal data; these are things like "workclass" or "marital status". It is my understanding it is best to convert nominal data like this into ...
1
vote
1answer
2k views

How to Replace an object in Pandas array using replace with dictionary from excel file?

suppose i have a data like this and i want to change the value of name and gender into an integer, and i have a dictionary like this i'm understand that i can use replace function in pandas, but i ...
0
votes
1answer
204 views

What is the best way to normalize histogram vectors to get distribution?

l have the following sample of 4 vectors of dimension 5 . They are sparse vectors, in a way that each value in a vector represent the frequency (number of occurrence of values). For instance v_1=[0,4,...
0
votes
1answer
2k views

Better input for Doc2Vec

I want to perform Doc2Vec on a twitter dataset. As each tweet consists of a nummber of special characters ,numbers, urls, mentions and hashtags, non-english words, what should be my input for Doc2Vec? ...
2
votes
1answer
54 views

Dealing with a dataset where a subset of points live in a higher dimensional space

I have a classification problem where I am dealing with economic data in a high dimensional (~400) space which includes dates, addresses, salaries, and a number of other variables. Most of the ...
0
votes
1answer
430 views

Need a Work-around for OneHotEncoder Issue in SKLearn Preprocessing

So, it seems that OneHotEncoder won't work with the np.int64 datatype (only np.int32)! Here's a sample of code: import numpy as np import pandas as pd from sklearn.preprocessing import ...
1
vote
2answers
286 views

Pre-process data images before training OneClassSVM and decrease number of features

I want to train a OneClassSVM() using sklearn, and I have a set of around 800 images in my training set. I am using opencv to read the images and resize them to constant dimensions (960x540) and then ...
-1
votes
1answer
45 views

Dataset processing question

I am a newbie in Data Science, so please don't blame me for stupid questions. Here is my problem. I've got a dataset (no empty cells, only numerical values) consisted of 15 columns (1st - user id, ...
-1
votes
1answer
528 views

How to preprocess data?

What is the generic way to preprocess data for machine learning and predictive models ?What are the sequence of steps to be taken?
2
votes
1answer
593 views

How Box cox and other transformations convert data into Normal Distributions?

How Box cox and other transformations convert data into Normal Distributions ?
1
vote
2answers
1k views

Normalizing time data

If I have a dataset with events occuring at certain times of day, Hour, how would I go about using this for, say, a classifier? Example: ...
1
vote
2answers
594 views

Is my normalization off?

So I have a DataFrame which consists of order data from customer on an exchange. I have a column Dollars which is the dollar ...
1
vote
0answers
24 views

Are annotated audio datasets augmented with mutated versions the way image datasets are?

Data augmentation is very standard for annotated image datasets for tasks like image labelling. Images are flipped, rotated, pixelated and so on, to add more training data and make the system robust ...
4
votes
6answers
11k views

issue with oneHotEncoding

So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course ...
1
vote
0answers
115 views

Chunker/shallow parser for spoken language

I'm trying to extract NPs from transcribed spoken text, such as um it's the bl- it's the blue one in the right no left hand corner which contains e.g. fillers (e.g. um) and disfluencies (e.g. bl-,...
1
vote
1answer
6k views

How to use the same scale with new data? - scikit learn - scikit learn

How do I use the same scale used in preprocessing with new data. Actual code: ...
-1
votes
1answer
345 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
-1
votes
1answer
157 views

Tableau: How to change multiple field names

From SQL server I imported multiple tables that each have multiple fields. Unfortunately the field names are not that descriptive (data is originally from SAP) but I have a separate Excel file that ...
2
votes
2answers
2k views

Keras loading images in incorrect format

So I was working with the the vgg16 model for dogs vs cats classification and I noticed that keras is not loading images in correct color format. The code is as follows: ...
2
votes
2answers
313 views

How to preprocess Acoustic Data

I am dealing with acoustic data with very high sampling frequency of 2MHz and want to build a classifier. I was wondering if there are any rules of thumb for preprocessing acoustic data. Is it better ...
-1
votes
2answers
990 views

Best way to tokenize tweet

While working with Twitter datasets, one thing that always confuses me is, How to tokenize the tweets. I have seen different open-source implementations using different schemes for tokenization. ...
3
votes
2answers
356 views

What pre processing should I use on data to feed into a CNN?

I have a dataset of shape 105 x 501 x 266 where 105 is the number of data and 501 x 266 is ...
4
votes
1answer
314 views

Should I standardize first or generate polynomials first?

Recently I am dealing a classification problem with some algorithms, say logistic regression. When I preprocess my data, I standardize all my features and then generate polynomial features based on ...
1
vote
0answers
162 views

how is countvectorizer used in real production environment?

how is countvectorizer used in real production environment? do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?...
-1
votes
1answer
100 views

Do I need equal number of bins for all attributes?

I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize ...
2
votes
2answers
69 views

How to use a dataset where attribute names are changed?

I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed. My goal is to use dataset ...
0
votes
2answers
120 views

Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)

I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following: ...
0
votes
1answer
110 views

Is pre-processing always neccessary?

I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary. I have a lot of ...
-1
votes
1answer
103 views

How to train Matlab on a range of IP addresses?

I'd like to train a Decision Tree using the Classification Learner App. I have a range of IP addresses, and a country that the IP address range belongs to. ...
7
votes
1answer
1k views

Extracting individual emails from an email thread

Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email ...
-1
votes
2answers
65 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
5
votes
2answers
2k views

Nested features with one to many relationships

I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level. For example, consider predicting machine failure based on its last ...
2
votes
2answers
3k views

Convert exponential to normal distribution

For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better ...
-1
votes
1answer
370 views

Image Classification, Convolution Network and Gamma Correction for images

I am working with NIST Special Database 4, which is a database of fingerprints. The objective is to train a convolutional network (CNN) and train it to classify the fingerprints. After I looked ...
1
vote
0answers
96 views

Transformation of Dependent and Independent Variables

I have a few Independent variables that's normal and a Dependent variables that's skewed , I pick log(feature+SHIFT) to correct skewness. The procedure I follow to get prediction is just take exp(...
2
votes
1answer
4k views

Real time noise removal using Savitzky-Golay Method

I would like to ask if Savitzky-Golay can be implemented on real-time data. I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone ...
0
votes
1answer
106 views

Data preprocessing, relative scale problems in features of same type

I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example. ...
1
vote
2answers
875 views

What are recommended ways\tools for processing large data from Excel Files?

A Very Happy New Year! I'm currently working on an analytics project with large volumes of data stored in excel files (about 50GB in 1000 files). The files use a custom formatting to store date-time ...
0
votes
1answer
47 views

Outliers Approach

Having a schema which the majority of the values are IDs. Like this example (this isn't my real data): ...
0
votes
1answer
535 views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...
5
votes
1answer
6k views

How to implement global contrast normalization in python?

I try to implement global contrast normalization in python from Yoshua Bengio's deep learning book. From the book, to get normalized image using global contrast normalization we use this equation $$\...
1
vote
2answers
419 views

Preprocessing to-be-predicted data in ML with R - “learn” and “apply” features

I have studied the usual preprocessing methods for Machine Learning but I couldn't cope the following specific problem. I apply the "usual" preparation for modeling (dummy variables, normalization, ...
2
votes
3answers
176 views

What are some method for pre-processing data in OCR?

I have a dataset for a supervised learning task. Each row is a vector with a value of pixmap value in a range [0,255] of gray colormap, each vector is labeled with a character. I have to assign each ...
1
vote
2answers
825 views

Binary classification: best ways to pre-procees the data

About the dataset I have a training dataset of 129 columns(last column being the classes, i.e., y values) 6068 rows I have to train some algo to do binary classification. The data set has 701 ...
1
vote
0answers
480 views

Clustering bitcoin addresses with k-means - how would one prepare input

I am looking to perform clustering on bitcoin addresses within the blockchain. I have generated a graph structure of the blockchain with a source address, destination address, value of transaction, in-...
1
vote
0answers
53 views

How do I convert a series of timestamps in seconds to milliseconds in order to distribute them smoothly

I have a script that collects memory usage data from a device running a Linux stack. I collect samples roughly every 200ms and write the measurement to a csv file with a timestamp. The problem is ...
3
votes
2answers
1k views

Machine Learning or Survival Analysis?

I am working on building prediction model for disk failures (time taken to occur a disk failure and what parameters could strongly affect disk failures). I am bit confused on- What data preprocessing ...
1
vote
1answer
3k views

What does normalizing and mean centering data do?

Are there any concerns to normalizing data to be within the range 0 - 1 and mean centering the data as well? Does it matter which comes first? If you do one, is the other not required?
1
vote
1answer
585 views

Should we convert independent continous variables (features) to categorical variable before using decision tree like classifier?

Consider I have one dependent variable to predict 'Attitude' which can take three values 'Positive/Negative/Neutral'. I have following independent variables or features- Age, Height, Gender, Income ...
5
votes
3answers
486 views

Do you apply outlier detection of numerical data in practical applications?

In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data. Do you apply outlier detection in your ...