Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-1
votes
1answer
101 views

How to train Matlab on a range of IP addresses?

I'd like to train a Decision Tree using the Classification Learner App. I have a range of IP addresses, and a country that the IP address range belongs to. ...
-1
votes
2answers
65 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
-1
votes
1answer
95 views

Do I need equal number of bins for all attributes?

I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize ...
4
votes
1answer
281 views

Should I standardize first or generate polynomials first?

Recently I am dealing a classification problem with some algorithms, say logistic regression. When I preprocess my data, I standardize all my features and then generate polynomial features based on ...
1
vote
0answers
156 views

how is countvectorizer used in real production environment?

how is countvectorizer used in real production environment? do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?...
2
votes
2answers
68 views

How to use a dataset where attribute names are changed?

I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed. My goal is to use dataset ...
0
votes
2answers
112 views

Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)

I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following: ...
0
votes
1answer
107 views

Is pre-processing always neccessary?

I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary. I have a lot of ...
1
vote
2answers
756 views

Binary classification: best ways to pre-procees the data

About the dataset I have a training dataset of 129 columns(last column being the classes, i.e., y values) 6068 rows I have to train some algo to do binary classification. The data set has 701 ...
-1
votes
1answer
333 views

Image Classification, Convolution Network and Gamma Correction for images

I am working with NIST Special Database 4, which is a database of fingerprints. The objective is to train a convolutional network (CNN) and train it to classify the fingerprints. After I looked ...
1
vote
0answers
90 views

Transformation of Dependent and Independent Variables

I have a few Independent variables that's normal and a Dependent variables that's skewed , I pick log(feature+SHIFT) to correct skewness. The procedure I follow to get prediction is just take exp(...
0
votes
1answer
97 views

Data preprocessing, relative scale problems in features of same type

I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example. ...
1
vote
2answers
846 views

What are recommended ways\tools for processing large data from Excel Files?

A Very Happy New Year! I'm currently working on an analytics project with large volumes of data stored in excel files (about 50GB in 1000 files). The files use a custom formatting to store date-time ...
0
votes
1answer
46 views

Outliers Approach

Having a schema which the majority of the values are IDs. Like this example (this isn't my real data): ...
0
votes
1answer
387 views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...
3
votes
2answers
1k views

Machine Learning or Survival Analysis?

I am working on building prediction model for disk failures (time taken to occur a disk failure and what parameters could strongly affect disk failures). I am bit confused on- What data preprocessing ...
5
votes
3answers
475 views

Do you apply outlier detection of numerical data in practical applications?

In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data. Do you apply outlier detection in your ...
9
votes
1answer
3k views

How to approach the numer.ai competition with anonymous scaled numerical predictors?

Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web. The system has changed from time to time and the set-up today is the following: ...
1
vote
1answer
544 views

Should we convert independent continous variables (features) to categorical variable before using decision tree like classifier?

Consider I have one dependent variable to predict 'Attitude' which can take three values 'Positive/Negative/Neutral'. I have following independent variables or features- Age, Height, Gender, Income ...
2
votes
1answer
838 views

Redundancy - is it a big problem?

I am trying to create a sentiment analysis program which will classify some of the tweets which i have collected under a hashtag. There are 7750 tweets in the dataset and I am labeling them into the ...
1
vote
0answers
319 views

How to best represent rate or proportion as a feature?

Given two features in a data set: NFObs (Number of failed observations) and NObs (Number of total observations), I'd like to ...
1
vote
1answer
56 views

Apply function on every four rows

Let's say I have a dataset like thisX<-matrix(rnorm(30), nrow=100, ncol=6). I am trying to find a way to apply the sum function to every four rows of column 3. ...
0
votes
1answer
85 views

Problem in constructing co-occurence matrix

I have a basic doubt in construction of co-occurence matrix from text. Coocurence matrix X is defined as : Xij : number of times word i occurs in context of word j in a window of size w. Now is this ...
4
votes
2answers
358 views

How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is ...
1
vote
1answer
386 views

What metrics must i use in my data(unstructured) preprocessing research?

i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats). i wish to prove preprocessing improves the content quality. are there metrics available to prove ...
3
votes
1answer
1k views

Denoising Autoenoders with Variable Length Input

I'm working on a problem with data from a continuous real-valued signal. The goal is to use ML to smooth the signal based off of past data. To accomplish this, the signal is windowed into a period ...
2
votes
0answers
292 views

User activity representation for Prediction/ML

I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data. The raw data is a sequence of all activities by all users: ...
2
votes
1answer
391 views

How should clickstream data be prepared before user segmentation can be performed?

I'm interested in doing segmentation/clustering of users in clickstream data and am looking for some good suggestions about how to go about it. Lets say my data consists of observations made up of ...
3
votes
0answers
103 views

Is it common to preprocess image data before sending it through a deep net? [duplicate]

I'm curious as how convolutional neural network are used in practice for object recognition. Is it common to perform data preprocessing before providing the data to the input layer ? If so, what types ...
2
votes
1answer
461 views

How can I preprocess multi-page image inputs in a theano/lasagne network?

I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no photographs or icons), and different ...
1
vote
2answers
465 views

Data preprocessing : Aggregation, feature creation, or else?

I have a problem to name data processing step. I have an attribute that contain string or null. I want to change the record of an attribute to 0 if null and 1 if not null. What preprocessing step ...
6
votes
2answers
194 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
2
votes
1answer
6k views

What is a benchmark model?

I am working on a breast cancer dataset (http://kdd.org/kdd-cup/view/kdd-cup-2008). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing. A ...
2
votes
1answer
129 views

Does it makes sense to apply feature scaling on timestamp

I was wondering if it makes sense to apply normal standardization on a feature like timestamp ? The data that I process are network packets. Thank you
3
votes
1answer
515 views

How to visualize data of a multidimensional dataset (TIMIT)

I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 ...
2
votes
2answers
249 views

Sampling for multi categorical variable

My hypothesis h depends on multiple categorical variables (a,b,c) each with their corresponding set of possible values ...
5
votes
1answer
8k views

How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

I want to use some Decision Tree learning, such as the Random Forest classifier. I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to ...