Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
4
votes
1answer
76 views

Feature importance over a subset of instance space instead of an entire instance space

I'm really curious if anyone has faced this problem before, or is it even widely studied at all. Imagine we have a feature that isn't important (based on many widely available and textbook feature ...
0
votes
2answers
64 views

What to do if my target variable is column of lists? [closed]

How I can transform my target variable(Y)? As it is list, I cann`t use it for fitting model, because I must use integers for fitting.
0
votes
2answers
812 views

How to detect phrases from an English sentence.

The question is not about detecting keyphrases. It is about detecting a combination of words makes a valid phrase or not. For example, "John reads New York Times in New York." Here, the phrases ...
3
votes
1answer
175 views

Normalizing test data

I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the ...
0
votes
1answer
144 views

Different approaches of creating the test set

I came across different approaches to creating a test set. Theoretically, it's quite simple, just pick some instances randomly, typically 20% of the dataset and set them aside. Below are the ...
1
vote
1answer
80 views

Dropping less frequently used categorical data?

I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite ...
3
votes
1answer
215 views

Standard correlation coefficient of various datasets

I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure 1. Why is it so ?. Even the shapes differ for ...
1
vote
3answers
67 views

Need help in understanding a small example

Pardon me, I agree the title of the question is not clear. I would like to know the understanding of below steps which are picked from the textbook "Hands on machine learning". ...
0
votes
1answer
23 views

Computer Vision: Handling dataset(3D data or scan) with different timesteps

I'm planning on training a CNN on CT scans for classification. The problem is CT scans are taken slice by slice, and in a typical scan, there could be more than 200 slices. The number of slices in a ...
1
vote
1answer
48 views

Why is it important to have sufficient number of instances in your dataset for each stratum?

As per the figure 1, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \$60,000. I didn't understand the explanation behind why housing['...
1
vote
1answer
65 views

How to check if audio samples have only noise or are silent?

I have a wav file I want to split into frames in order to feed it into a machine learning model. The problem is that the audio has silence with some noise at some points. My problem is that I do not ...
3
votes
2answers
56 views

Normalising data with multiple methods

When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score ...
0
votes
1answer
3k views

Extract all the data of a particular month from dataset of different years

In a dataset containing temperatures of different years and I want to extract data of particular month from all different years in single liner code what is the syntax? To extract all the data from a ...
2
votes
2answers
1k views

How to scale prediction back after preprocessing

So I'm a newbie to machine learning and am currently using the iris data set. I ran through a quick online tutorial about predicting stock prices and thought I'd try and do the iris one myself. The ...
2
votes
1answer
318 views

normalization/denormalization for linear regression problem

My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after ...
1
vote
0answers
182 views

Delete strings with a specific last character in tibco spotfire [closed]

I am using tibco spotfire and I have columns with data like this "0001V", "0002V", "0003N". I know data with the 'N' at the end is not valid so I need to delete the numbers which have that. I think I ...
1
vote
1answer
737 views

Boruta Feature Selection package

I am using Boruta feature selection (with Random forest) to decide the important features in the below data set. ...
2
votes
1answer
241 views

How can I measure the similarity between 2 IP addresses? Is there any code to re-use?

I need to measure the similarity of 2 IP addresses. I could not find any sample code in scala or other languages to find the distance between 2 IPs.
0
votes
1answer
50 views

Removing irrelevant content in columns

I have a dataset with a column named Package Size and it has data in two units inches and meters mentioned in words. How do I preprocess the data such that I remove the unnecessary word and also ...
11
votes
3answers
12k views

Image resizing and padding for CNN

I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized ...
1
vote
1answer
83 views

standardize dataset with numerical and dummy features

I have a dataset with both numerical and categorical features (variables), I converted all the categorical variables in dummies, then I split the train and test data. Now I am at the step where I ...
2
votes
1answer
438 views

How to load a csv file into [Pandas] dataframe if computer runs out of RAM?

I have been trying to train a neural network, but my computer is always running out of RAM memory when I'm loading the dataframe with Pandas. Its a .csv file that is like 7+ GB. I wanted to try some ...
3
votes
1answer
118 views

Video classification of birds

I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked ...
0
votes
1answer
46 views

Image to image translation from image to sketch using GANs, do I need to annotated the face at all?

So far I have read 2 papers on GANs and it seems like they are unsupervised networks that only uses supervision for the discriminator. If I want to translate an image from a picture to line art all I ...
1
vote
1answer
136 views

Extract relevant vocabulary from a document

I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g: ...
0
votes
1answer
51 views

Which cloud platform to maximize my impact as a data scientist? [closed]

I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer. By this I mean handling the following on my own: preprocess big data at low ...
2
votes
2answers
5k views

One Hot Encoding vs Word Embeding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
2
votes
2answers
2k views

How to check and correct misspelling in the data of pairs of words?

I have user generated text containing names of ports often containing typos and the actual port names. I would like to correct the misspelling of user generated text containing the names of ports. Can ...
1
vote
0answers
21 views

Variable TimeStep in Spatio Temporal 3DConvNet

I'm working on a new project on climate data. I suppose my output $Y(H*W*1)$ is a consequence of a given data scenario $X(C=4*T*H*W)$ and an initial state $Y0(H*W*1)$. I've chosen a temporal ...
0
votes
2answers
194 views

Filling in missing values

Suppose I have a train and test data set and both of them contain missing values. Can I join both of them and then do fillna.(alldata.mean()) or will that cause ...
0
votes
1answer
37 views

Should I remove features that occur very rarely to build a model?

I am trying ML techniques in language processing. I have got 3000 short texts and I extract features(words and phrases) from all of them and build a vocabulary. I end up with 6000 od features and most ...
0
votes
2answers
48 views

Why is it necessary to clean/preprocess data if the unseen real data is bound to consist missing values, outliers etc.?

Can it really be called Generalization if we remove/modify data points to suit our model?
0
votes
1answer
571 views

How to treat the datasets with unreliable labels

I have a dataset belonging to three different classes: A, B and C. Among these three classes, the classification for label C is unreliable comparing to other two classes. In other words, some of the ...
0
votes
2answers
165 views

Data binning - Why we need to transform Categorical Variables?

Having a lot of categorical features and other numerics why we need to transform the categorical to binary values? Is it for using the values in mathematics functions of the algorithms? Thanks!
0
votes
1answer
31 views

Binning which variables?

I will try to implement a k-means algorithm over this dataset: ...
1
vote
1answer
185 views

Exploratory Data Analysis and selecting good predictor variables ?

In what way would exploratory data analysis aid in feature selection, other than to preprocess the data ? Say, if a bivariate analysis was conducted for each predictor variable w.r.t. the target ...
10
votes
4answers
8k views

Different Test Set and Training Set Distribution

I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test ...
5
votes
2answers
4k views

Why is input preprocessing in VGG16 in Keras not 1/255.0

I am just trying to use pre-trained vgg16 to make prediction in Keras like this. ...
2
votes
1answer
55 views

Preprocess list data

I got question about preparation data for my ML algorithm. Raw data has format similar to: ...
4
votes
3answers
6k views

How to get spike values from a value sequence?

I have pile of vectors where the values could be plotted like this: Now I want to extract the "spike values" (over a certain threshold say 15,000). In this case there is fifteen. How could this be ...
6
votes
1answer
4k views

Data preprocessing: Should we normalise images pixel-wise?

Let me present you with a toy example and a reasoning on image normalisation I had: Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (...
0
votes
1answer
253 views

Preparing data for a live prediction engine

I need to build a live prediction engine (either using Ensemble methods/ RNN/ Keras Classifier) that could learn from historical data what kinds of users are likely to transact and, in real-time, ...
2
votes
1answer
1k views

Should I Impute target values?

I am new to data science and I am currently playing around a bit. Data exploration and preparation is really annoying. Eventhough I use pandas. I achieved imputing missing values in independant ...
3
votes
1answer
2k views

One hot encoding of target space

I had a face to face interview for a data scientist job a few days ago. One of the questions I was asked was: in the case of classifier predicting the brand of TV from some features (price, size, ...
1
vote
2answers
199 views

Should I apply PCA on the entire dataset or just the nominal values?

I have a data-set with 14~ attributes, roughly half of them nominal. I've used a binary vectorizer to convert these values to a number of attributes. The number of attributes, naturally, ballooned up; ...
0
votes
1answer
513 views

Converting nominal data to numeric - is using dictionaries the right approach?

Currently I have a data-set with roughly 7 types of nominal data; these are things like "workclass" or "marital status". It is my understanding it is best to convert nominal data like this into ...
1
vote
1answer
2k views

How to Replace an object in Pandas array using replace with dictionary from excel file?

suppose i have a data like this and i want to change the value of name and gender into an integer, and i have a dictionary like this i'm understand that i can use replace function in pandas, but i ...
0
votes
1answer
193 views

What is the best way to normalize histogram vectors to get distribution?

l have the following sample of 4 vectors of dimension 5 . They are sparse vectors, in a way that each value in a vector represent the frequency (number of occurrence of values). For instance v_1=[0,4,...
0
votes
1answer
2k views

Better input for Doc2Vec

I want to perform Doc2Vec on a twitter dataset. As each tweet consists of a nummber of special characters ,numbers, urls, mentions and hashtags, non-english words, what should be my input for Doc2Vec? ...
2
votes
1answer
52 views

Dealing with a dataset where a subset of points live in a higher dimensional space

I have a classification problem where I am dealing with economic data in a high dimensional (~400) space which includes dates, addresses, salaries, and a number of other variables. Most of the ...