Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
1
vote
2answers
2k views

Normalizing time data

If I have a dataset with events occuring at certain times of day, Hour, how would I go about using this for, say, a classifier? Example: ...
1
vote
0answers
24 views

Are annotated audio datasets augmented with mutated versions the way image datasets are?

Data augmentation is very standard for annotated image datasets for tasks like image labelling. Images are flipped, rotated, pixelated and so on, to add more training data and make the system robust ...
2
votes
1answer
6k views

Real time noise removal using Savitzky-Golay Method

I would like to ask if Savitzky-Golay can be implemented on real-time data. I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone ...
1
vote
1answer
7k views

How to use the same scale with new data? - scikit learn - scikit learn

How do I use the same scale used in preprocessing with new data. Actual code: ...
1
vote
0answers
123 views

Chunker/shallow parser for spoken language

I'm trying to extract NPs from transcribed spoken text, such as um it's the bl- it's the blue one in the right no left hand corner which contains e.g. fillers (e.g. um) and disfluencies (e.g. bl-,...
-1
votes
1answer
399 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
1
vote
2answers
637 views

Preprocessing to-be-predicted data in ML with R - “learn” and “apply” features

I have studied the usual preprocessing methods for Machine Learning but I couldn't cope the following specific problem. I apply the "usual" preparation for modeling (dummy variables, normalization, ...
2
votes
3answers
573 views

Preprocessing in Data mining?

I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the ...
-1
votes
2answers
2k views

Best way to tokenize tweet

While working with Twitter datasets, one thing that always confuses me is, How to tokenize the tweets. I have seen different open-source implementations using different schemes for tokenization. ...
-1
votes
1answer
323 views

Tableau: How to change multiple field names

From SQL server I imported multiple tables that each have multiple fields. Unfortunately the field names are not that descriptive (data is originally from SAP) but I have a separate Excel file that ...
2
votes
2answers
2k views

Keras loading images in incorrect format

So I was working with the the vgg16 model for dogs vs cats classification and I noticed that keras is not loading images in correct color format. The code is as follows: ...
2
votes
2answers
356 views

How to preprocess Acoustic Data

I am dealing with acoustic data with very high sampling frequency of 2MHz and want to build a classifier. I was wondering if there are any rules of thumb for preprocessing acoustic data. Is it better ...
-1
votes
1answer
112 views

How to train Matlab on a range of IP addresses?

I'd like to train a Decision Tree using the Classification Learner App. I have a range of IP addresses, and a country that the IP address range belongs to. ...
-1
votes
2answers
71 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
-1
votes
1answer
120 views

Do I need equal number of bins for all attributes?

I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize ...
5
votes
1answer
682 views

Should I standardize first or generate polynomials first?

Recently I am dealing a classification problem with some algorithms, say logistic regression. When I preprocess my data, I standardize all my features and then generate polynomial features based on ...
2
votes
0answers
177 views

how is countvectorizer used in real production environment?

how is countvectorizer used in real production environment? do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?...
2
votes
2answers
78 views

How to use a dataset where attribute names are changed?

I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed. My goal is to use dataset ...
0
votes
2answers
142 views

Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)

I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following: ...
0
votes
1answer
160 views

Is pre-processing always neccessary?

I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary. I have a lot of ...
-1
votes
1answer
489 views

Image Classification, Convolution Network and Gamma Correction for images

I am working with NIST Special Database 4, which is a database of fingerprints. The objective is to train a convolutional network (CNN) and train it to classify the fingerprints. After I looked ...
1
vote
0answers
128 views

Transformation of Dependent and Independent Variables

I have a few Independent variables that's normal and a Dependent variables that's skewed , I pick log(feature+SHIFT) to correct skewness. The procedure I follow to get prediction is just take exp(...
0
votes
1answer
119 views

Data preprocessing, relative scale problems in features of same type

I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example. ...
1
vote
2answers
1k views

What are recommended ways\tools for processing large data from Excel Files?

A Very Happy New Year! I'm currently working on an analytics project with large volumes of data stored in excel files (about 50GB in 1000 files). The files use a custom formatting to store date-time ...
0
votes
1answer
57 views

Outliers Approach

Having a schema which the majority of the values are IDs. Like this example (this isn't my real data): ...
1
vote
1answer
1k views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...
3
votes
2answers
2k views

Machine Learning or Survival Analysis?

I am working on building prediction model for disk failures (time taken to occur a disk failure and what parameters could strongly affect disk failures). I am bit confused on- What data preprocessing ...
5
votes
3answers
547 views

Do you apply outlier detection of numerical data in practical applications?

In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data. Do you apply outlier detection in your ...
9
votes
1answer
4k views

How to approach the numer.ai competition with anonymous scaled numerical predictors?

Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web. The system has changed from time to time and the set-up today is the following: ...
1
vote
1answer
753 views

Should we convert independent continous variables (features) to categorical variable before using decision tree like classifier?

Consider I have one dependent variable to predict 'Attitude' which can take three values 'Positive/Negative/Neutral'. I have following independent variables or features- Age, Height, Gender, Income ...
3
votes
1answer
1k views

Redundancy - is it a big problem?

I am trying to create a sentiment analysis program which will classify some of the tweets which i have collected under a hashtag. There are 7750 tweets in the dataset and I am labeling them into the ...
1
vote
0answers
407 views

How to best represent rate or proportion as a feature?

Given two features in a data set: NFObs (Number of failed observations) and NObs (Number of total observations), I'd like to ...
1
vote
1answer
62 views

Apply function on every four rows

Let's say I have a dataset like thisX<-matrix(rnorm(30), nrow=100, ncol=6). I am trying to find a way to apply the sum function to every four rows of column 3. ...
0
votes
1answer
98 views

Problem in constructing co-occurence matrix

I have a basic doubt in construction of co-occurence matrix from text. Coocurence matrix X is defined as : Xij : number of times word i occurs in context of word j in a window of size w. Now is this ...
4
votes
2answers
414 views

How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is ...
3
votes
1answer
2k views

Denoising Autoenoders with Variable Length Input

I'm working on a problem with data from a continuous real-valued signal. The goal is to use ML to smooth the signal based off of past data. To accomplish this, the signal is windowed into a period ...
2
votes
0answers
314 views

User activity representation for Prediction/ML

I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data. The raw data is a sequence of all activities by all users: ...
2
votes
1answer
488 views

How should clickstream data be prepared before user segmentation can be performed?

I'm interested in doing segmentation/clustering of users in clickstream data and am looking for some good suggestions about how to go about it. Lets say my data consists of observations made up of ...
3
votes
0answers
107 views

Is it common to preprocess image data before sending it through a deep net? [duplicate]

I'm curious as how convolutional neural network are used in practice for object recognition. Is it common to perform data preprocessing before providing the data to the input layer ? If so, what types ...
2
votes
1answer
483 views

How can I preprocess multi-page image inputs in a theano/lasagne network?

I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no photographs or icons), and different ...
1
vote
2answers
557 views

Data preprocessing : Aggregation, feature creation, or else?

I have a problem to name data processing step. I have an attribute that contain string or null. I want to change the record of an attribute to 0 if null and 1 if not null. What preprocessing step ...
6
votes
2answers
202 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
5
votes
1answer
10k views

What is a benchmark model?

I am working on a breast cancer dataset (http://kdd.org/kdd-cup/view/kdd-cup-2008). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing. A ...
2
votes
1answer
179 views

Does it makes sense to apply feature scaling on timestamp

I was wondering if it makes sense to apply normal standardization on a feature like timestamp ? The data that I process are network packets. Thank you
3
votes
1answer
669 views

How to visualize data of a multidimensional dataset (TIMIT)

I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 ...
3
votes
2answers
572 views

Sampling for multi categorical variable

My hypothesis h depends on multiple categorical variables (a,b,c) each with their corresponding set of possible values ...
7
votes
1answer
10k views

How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

I want to use some Decision Tree learning, such as the Random Forest classifier. I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to ...

1
4 5 6 7
8