Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
4
votes
3answers
11k views

How to get spike values from a value sequence?

I have pile of vectors where the values could be plotted like this: Now I want to extract the "spike values" (over a certain threshold say 15,000). In this case there is fifteen. How could this be ...
2
votes
1answer
344 views

How can I measure the similarity between 2 IP addresses? Is there any code to re-use?

I need to measure the similarity of 2 IP addresses. I could not find any sample code in scala or other languages to find the distance between 2 IPs.
0
votes
1answer
186 views

What is the the cost of combining categorical variables?

I have 2 categorical variables e.g. state and city. Missing are only in city. As opposed to throwing out all observations with missing values for city or throwing out city all together I was ...
0
votes
1answer
68 views

Does discretization of continuous features also lose information about distance?

During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way. But doesn't it also lose information about distances of features? For example, if we ...
0
votes
1answer
57 views

Is it correct to use non-target values of test set to engineer new features for train set?

Suppose, I have a dataset with a feature_1 value and a target value. Now, I want to engineer a new feature by creating relative value by subtracting mean from each value. Question: Can I (1) use ...
1
vote
0answers
71 views

How to deal with attributes that can vary arbitrarily for each sample?

Let's say we are trying to classify cars into five different categories. For this, we have a lot of samples described by color, brand, model, year of manufacture and so on. For instance, imagine ...
5
votes
2answers
8k views

Why is input preprocessing in VGG16 in Keras not 1/255.0

I am just trying to use pre-trained vgg16 to make prediction in Keras like this. ...
1
vote
0answers
278 views

Audio signal signal processing for background noise reduction or removal

I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and ...
0
votes
1answer
613 views

pandas: How to impute the categorical column by the nearest neighbors?

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...
0
votes
1answer
49 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
1
vote
0answers
83 views

Any tool that can help on manually label a time series data please?

i am working with a 10 years weekly financial time series data set, which has the standard format date open high low close volume. i would like to manually label (classify) the data set with label/...
1
vote
0answers
282 views

How standardizing and/or log transformation affect prediction result in machine learning models

I recently ran an elastic net model on my data. My predictors are mostly skewed. I found my model perform slightly better when I standardize on log-transformed data than standardizing on original data....
1
vote
0answers
158 views

Lowercase texts before tokenizing as pre-processing step for alignment

I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I ...
1
vote
0answers
196 views

Transposing a dataframe (time-series)

I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting): ...
1
vote
2answers
249 views

Preprocessing of Sudoku Dataset from Kaggle

Dataset: https://www.kaggle.com/bryanpark/sudoku I would like to create a neural network for this Dataset. Feature: ...
1
vote
0answers
49 views

Normalize data with uneven groups?

I have a dataset with 3 independent variables [city, industry, amount] and wish to normalize the amount. But I wish to do it with respect to industry and city. Simply grouping by the city and industry ...
1
vote
0answers
13 views

Are there some resources for filters specifically applicable in big data applications? [closed]

Are there some resources for filters specifically applicable in big data applications? Particularly, are there major differences between filter design for other domains and filter design for data ...
0
votes
2answers
1k views

How to detect phrases from an English sentence.

The question is not about detecting keyphrases. It is about detecting a combination of words makes a valid phrase or not. For example, "John reads New York Times in New York." Here, the ...
1
vote
0answers
131 views

Input data for this dataset to be feed into keras for training

Suppose I have 3 csv files which forms the dataset for training a machine learning model in Keras. file1.csv ...
0
votes
2answers
197 views

Basics: What is the correct sequence for preparing simple data for ML?

I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one). I was just wondering what would be the best way to organise the data to avoid redundancy with the ...
0
votes
1answer
735 views

How to interpret Hashingvectorizer representation?

I cannot really understand the logic behind Hashingvectorizer for text feature extraction. I can follow the logic of Bag of Word or TFiDF where the features are values for all/certain words/N-grams ...
0
votes
2answers
75 views

What to do if my target variable is column of lists? [closed]

How I can transform my target variable(Y)? As it is list, I cann`t use it for fitting model, because I must use integers for fitting.
3
votes
1answer
348 views

Normalizing test data

I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the ...
1
vote
0answers
28 views

Variable TimeStep in Spatio Temporal 3DConvNet

I'm working on a new project on climate data. I suppose my output $Y(H*W*1)$ is a consequence of a given data scenario $X(C=4*T*H*W)$ and an initial state $Y0(H*W*1)$. I've chosen a temporal ...
1
vote
1answer
143 views

standardize dataset with numerical and dummy features

I have a dataset with both numerical and categorical features (variables), I converted all the categorical variables in dummies, then I split the train and test data. Now I am at the step where I ...
1
vote
1answer
147 views

Dropping less frequently used categorical data?

I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite ...
1
vote
3answers
121 views

Need help in understanding a small example

Pardon me, I agree the title of the question is not clear. I would like to know the understanding of below steps which are picked from the textbook "Hands on machine learning". ...
0
votes
1answer
27 views

Computer Vision: Handling dataset(3D data or scan) with different timesteps

I'm planning on training a CNN on CT scans for classification. The problem is CT scans are taken slice by slice, and in a typical scan, there could be more than 200 slices. The number of slices in a ...
1
vote
1answer
60 views

Why is it important to have sufficient number of instances in your dataset for each stratum?

As per the figure 1, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \$60,000. I didn't understand the explanation behind why housing['...
1
vote
1answer
119 views

How to check if audio samples have only noise or are silent?

I have a wav file I want to split into frames in order to feed it into a machine learning model. The problem is that the audio has silence with some noise at some points. My problem is that I do not ...
3
votes
2answers
3k views

How to scale prediction back after preprocessing

So I'm a newbie to machine learning and am currently using the iris data set. I ran through a quick online tutorial about predicting stock prices and thought I'd try and do the iris one myself. The ...
3
votes
2answers
433 views

What pre processing should I use on data to feed into a CNN?

I have a dataset of shape 105 x 501 x 266 where 105 is the number of data and 501 x 266 is ...
0
votes
2answers
55 views
4
votes
3answers
1k views

How would one separate digits for number recognition?

Assuming that one has a neural network capable of returning the numerical digit from a given image of size 28x28px. How would one split an image of unknown size and an unknown amount of digits into a ...
1
vote
0answers
236 views

Delete strings with a specific last character in tibco spotfire [closed]

I am using tibco spotfire and I have columns with data like this "0001V", "0002V", "0003N". I know data with the 'N' at the end is not valid so I need to delete the numbers which have that. I think I ...
1
vote
1answer
1k views

Boruta Feature Selection package

I am using Boruta feature selection (with Random forest) to decide the important features in the below data set. ...
6
votes
7answers
4k views

How to define a distance measure between two IP addresses?

I have IP addresses as feature and I would like to know how much two IP addresses are similar to each other to use the difference in an Euclidean distance measure (in order to quantify the ...
0
votes
1answer
93 views

Removing irrelevant content in columns

I have a dataset with a column named Package Size and it has data in two units inches and meters mentioned in words. How do I preprocess the data such that I remove the unnecessary word and also ...
2
votes
1answer
2k views

Should I Impute target values?

I am new to data science and I am currently playing around a bit. Data exploration and preparation is really annoying. Eventhough I use pandas. I achieved imputing missing values in independant ...
0
votes
2answers
208 views

Filling in missing values

Suppose I have a train and test data set and both of them contain missing values. Can I join both of them and then do fillna.(alldata.mean()) or will that cause ...
8
votes
1answer
6k views

Data preprocessing: Should we normalise images pixel-wise?

Let me present you with a toy example and a reasoning on image normalisation I had: Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (...
3
votes
1answer
170 views

Video classification of birds

I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked ...
0
votes
1answer
63 views

Image to image translation from image to sketch using GANs, do I need to annotated the face at all?

So far I have read 2 papers on GANs and it seems like they are unsupervised networks that only uses supervision for the discriminator. If I want to translate an image from a picture to line art all I ...
2
votes
1answer
536 views

How to load a csv file into [Pandas] dataframe if computer runs out of RAM?

I have been trying to train a neural network, but my computer is always running out of RAM memory when I'm loading the dataframe with Pandas. Its a .csv file that is like 7+ GB. I wanted to try some ...
40
votes
2answers
39k views

How to prepare/augment images for neural network?

I would like to use a neural network for image classification. I'll start with pre-trained CaffeNet and train it for my application. How should I prepare the input images? In this case, all the ...
1
vote
1answer
323 views

Extract relevant vocabulary from a document

I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g: ...
0
votes
1answer
53 views

Which cloud platform to maximize my impact as a data scientist? [closed]

I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer. By this I mean handling the following on my own: preprocess big data at low ...
2
votes
2answers
4k views

How to check and correct misspelling in the data of pairs of words?

I have user generated text containing names of ports often containing typos and the actual port names. I would like to correct the misspelling of user generated text containing the names of ports. Can ...
0
votes
1answer
40 views

Should I remove features that occur very rarely to build a model?

I am trying ML techniques in language processing. I have got 3000 short texts and I extract features(words and phrases) from all of them and build a vocabulary. I end up with 6000 od features and most ...
1
vote
1answer
1k views

How to treat the datasets with unreliable labels

I have a dataset belonging to three different classes: A, B and C. Among these three classes, the classification for label C is unreliable comparing to other two classes. In other words, some of the ...

1
3 4
5
6 7