Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
300 views

How to best represent rate or proportion as a feature?

Given two features in a data set: NFObs (Number of failed observations) and NObs (Number of total observations), I'd like to ...
1
vote
1answer
56 views

Apply function on every four rows

Let's say I have a dataset like thisX<-matrix(rnorm(30), nrow=100, ncol=6). I am trying to find a way to apply the sum function to every four rows of column 3. ...
0
votes
1answer
82 views

Problem in constructing co-occurence matrix

I have a basic doubt in construction of co-occurence matrix from text. Coocurence matrix X is defined as : Xij : number of times word i occurs in context of word j in a window of size w. Now is this ...
4
votes
2answers
351 views

How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is ...
1
vote
1answer
377 views

What metrics must i use in my data(unstructured) preprocessing research?

i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats). i wish to prove preprocessing improves the content quality. are there metrics available to prove ...
2
votes
0answers
289 views

User activity representation for Prediction/ML

I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data. The raw data is a sequence of all activities by all users: ...
3
votes
0answers
103 views

Is it common to preprocess image data before sending it through a deep net? [duplicate]

I'm curious as how convolutional neural network are used in practice for object recognition. Is it common to perform data preprocessing before providing the data to the input layer ? If so, what types ...
2
votes
1answer
460 views

How can I preprocess multi-page image inputs in a theano/lasagne network?

I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no photographs or icons), and different ...
1
vote
2answers
446 views

Data preprocessing : Aggregation, feature creation, or else?

I have a problem to name data processing step. I have an attribute that contain string or null. I want to change the record of an attribute to 0 if null and 1 if not null. What preprocessing step ...
6
votes
2answers
192 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
2
votes
1answer
6k views

What is a benchmark model?

I am working on a breast cancer dataset (http://kdd.org/kdd-cup/view/kdd-cup-2008). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing. A ...
5
votes
7answers
3k views

How to define a distance measure between two IP addresses?

I have IP addresses as feature and I would like to know how much two IP addresses are similar to each other to use the difference in an Euclidean distance measure (in order to quantify the ...
2
votes
1answer
127 views

Does it makes sense to apply feature scaling on timestamp

I was wondering if it makes sense to apply normal standardization on a feature like timestamp ? The data that I process are network packets. Thank you
3
votes
1answer
503 views

How to visualize data of a multidimensional dataset (TIMIT)

I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 ...
2
votes
2answers
210 views

Sampling for multi categorical variable

My hypothesis h depends on multiple categorical variables (a,b,c) each with their corresponding set of possible values ...
2
votes
1answer
376 views

How should clickstream data be prepared before user segmentation can be performed?

I'm interested in doing segmentation/clustering of users in clickstream data and am looking for some good suggestions about how to go about it. Lets say my data consists of observations made up of ...
1
vote
3answers
532 views

Preprocessing in Data mining?

I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the ...
3
votes
1answer
1k views

Denoising Autoenoders with Variable Length Input

I'm working on a problem with data from a continuous real-valued signal. The goal is to use ML to smooth the signal based off of past data. To accomplish this, the signal is windowed into a period ...
5
votes
1answer
8k views

How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

I want to use some Decision Tree learning, such as the Random Forest classifier. I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to ...
40
votes
2answers
37k views

How to prepare/augment images for neural network?

I would like to use a neural network for image classification. I'll start with pre-trained CaffeNet and train it for my application. How should I prepare the input images? In this case, all the ...