Questions tagged [preprocessing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
150 views

Extract relevant vocabulary from a document

I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g: ...
0
votes
1answer
51 views

Which cloud platform to maximize my impact as a data scientist? [closed]

I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer. By this I mean handling the following on my own: preprocess big data at low ...
2
votes
2answers
3k views

How to check and correct misspelling in the data of pairs of words?

I have user generated text containing names of ports often containing typos and the actual port names. I would like to correct the misspelling of user generated text containing the names of ports. Can ...
0
votes
1answer
37 views

Should I remove features that occur very rarely to build a model?

I am trying ML techniques in language processing. I have got 3000 short texts and I extract features(words and phrases) from all of them and build a vocabulary. I end up with 6000 od features and most ...
0
votes
1answer
618 views

How to treat the datasets with unreliable labels

I have a dataset belonging to three different classes: A, B and C. Among these three classes, the classification for label C is unreliable comparing to other two classes. In other words, some of the ...
0
votes
2answers
170 views

Data binning - Why we need to transform Categorical Variables?

Having a lot of categorical features and other numerics why we need to transform the categorical to binary values? Is it for using the values in mathematics functions of the algorithms? Thanks!
1
vote
1answer
192 views

Exploratory Data Analysis and selecting good predictor variables ?

In what way would exploratory data analysis aid in feature selection, other than to preprocess the data ? Say, if a bivariate analysis was conducted for each predictor variable w.r.t. the target ...
0
votes
1answer
31 views

Binning which variables?

I will try to implement a k-means algorithm over this dataset: ...
10
votes
4answers
9k views

Different Test Set and Training Set Distribution

I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test ...
1
vote
1answer
2k views

How to Replace an object in Pandas array using replace with dictionary from excel file?

suppose i have a data like this and i want to change the value of name and gender into an integer, and i have a dictionary like this i'm understand that i can use replace function in pandas, but i ...
0
votes
1answer
253 views

Preparing data for a live prediction engine

I need to build a live prediction engine (either using Ensemble methods/ RNN/ Keras Classifier) that could learn from historical data what kinds of users are likely to transact and, in real-time, ...
2
votes
1answer
56 views

Preprocess list data

I got question about preparation data for my ML algorithm. Raw data has format similar to: ...
3
votes
1answer
2k views

One hot encoding of target space

I had a face to face interview for a data scientist job a few days ago. One of the questions I was asked was: in the case of classifier predicting the brand of TV from some features (price, size, ...
1
vote
2answers
201 views

Should I apply PCA on the entire dataset or just the nominal values?

I have a data-set with 14~ attributes, roughly half of them nominal. I've used a binary vectorizer to convert these values to a number of attributes. The number of attributes, naturally, ballooned up; ...
0
votes
1answer
530 views

Converting nominal data to numeric - is using dictionaries the right approach?

Currently I have a data-set with roughly 7 types of nominal data; these are things like "workclass" or "marital status". It is my understanding it is best to convert nominal data like this into ...
2
votes
1answer
53 views

Dealing with a dataset where a subset of points live in a higher dimensional space

I have a classification problem where I am dealing with economic data in a high dimensional (~400) space which includes dates, addresses, salaries, and a number of other variables. Most of the ...
-1
votes
1answer
45 views

Dataset processing question

I am a newbie in Data Science, so please don't blame me for stupid questions. Here is my problem. I've got a dataset (no empty cells, only numerical values) consisted of 15 columns (1st - user id, ...
5
votes
2answers
1k views

Nested features with one to many relationships

I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level. For example, consider predicting machine failure based on its last ...
0
votes
1answer
195 views

What is the best way to normalize histogram vectors to get distribution?

l have the following sample of 4 vectors of dimension 5 . They are sparse vectors, in a way that each value in a vector represent the frequency (number of occurrence of values). For instance v_1=[0,4,...
0
votes
1answer
2k views

Better input for Doc2Vec

I want to perform Doc2Vec on a twitter dataset. As each tweet consists of a nummber of special characters ,numbers, urls, mentions and hashtags, non-english words, what should be my input for Doc2Vec? ...
2
votes
3answers
174 views

What are some method for pre-processing data in OCR?

I have a dataset for a supervised learning task. Each row is a vector with a value of pixmap value in a range [0,255] of gray colormap, each vector is labeled with a character. I have to assign each ...
0
votes
1answer
406 views

Need a Work-around for OneHotEncoder Issue in SKLearn Preprocessing

So, it seems that OneHotEncoder won't work with the np.int64 datatype (only np.int32)! Here's a sample of code: import numpy as np import pandas as pd from sklearn.preprocessing import ...
1
vote
0answers
50 views

How do I convert a series of timestamps in seconds to milliseconds in order to distribute them smoothly

I have a script that collects memory usage data from a device running a Linux stack. I collect samples roughly every 200ms and write the measurement to a csv file with a timestamp. The problem is ...
-1
votes
1answer
484 views

How to preprocess data?

What is the generic way to preprocess data for machine learning and predictive models ?What are the sequence of steps to be taken?
2
votes
1answer
562 views

How Box cox and other transformations convert data into Normal Distributions?

How Box cox and other transformations convert data into Normal Distributions ?
1
vote
2answers
584 views

Is my normalization off?

So I have a DataFrame which consists of order data from customer on an exchange. I have a column Dollars which is the dollar ...
1
vote
2answers
1k views

Normalizing time data

If I have a dataset with events occuring at certain times of day, Hour, how would I go about using this for, say, a classifier? Example: ...
1
vote
0answers
24 views

Are annotated audio datasets augmented with mutated versions the way image datasets are?

Data augmentation is very standard for annotated image datasets for tasks like image labelling. Images are flipped, rotated, pixelated and so on, to add more training data and make the system robust ...
2
votes
1answer
3k views

Real time noise removal using Savitzky-Golay Method

I would like to ask if Savitzky-Golay can be implemented on real-time data. I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone ...
1
vote
1answer
5k views

How to use the same scale with new data? - scikit learn - scikit learn

How do I use the same scale used in preprocessing with new data. Actual code: ...
1
vote
0answers
111 views

Chunker/shallow parser for spoken language

I'm trying to extract NPs from transcribed spoken text, such as um it's the bl- it's the blue one in the right no left hand corner which contains e.g. fillers (e.g. um) and disfluencies (e.g. bl-,...
-1
votes
1answer
316 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
1
vote
2answers
389 views

Preprocessing to-be-predicted data in ML with R - “learn” and “apply” features

I have studied the usual preprocessing methods for Machine Learning but I couldn't cope the following specific problem. I apply the "usual" preparation for modeling (dummy variables, normalization, ...
1
vote
3answers
533 views

Preprocessing in Data mining?

I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the ...
-1
votes
2answers
881 views

Best way to tokenize tweet

While working with Twitter datasets, one thing that always confuses me is, How to tokenize the tweets. I have seen different open-source implementations using different schemes for tokenization. ...
-1
votes
1answer
125 views

Tableau: How to change multiple field names

From SQL server I imported multiple tables that each have multiple fields. Unfortunately the field names are not that descriptive (data is originally from SAP) but I have a separate Excel file that ...
2
votes
2answers
2k views

Keras loading images in incorrect format

So I was working with the the vgg16 model for dogs vs cats classification and I noticed that keras is not loading images in correct color format. The code is as follows: ...
2
votes
2answers
280 views

How to preprocess Acoustic Data

I am dealing with acoustic data with very high sampling frequency of 2MHz and want to build a classifier. I was wondering if there are any rules of thumb for preprocessing acoustic data. Is it better ...
-1
votes
1answer
100 views

How to train Matlab on a range of IP addresses?

I'd like to train a Decision Tree using the Classification Learner App. I have a range of IP addresses, and a country that the IP address range belongs to. ...
-1
votes
2answers
65 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
-1
votes
1answer
93 views

Do I need equal number of bins for all attributes?

I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize ...
4
votes
1answer
269 views

Should I standardize first or generate polynomials first?

Recently I am dealing a classification problem with some algorithms, say logistic regression. When I preprocess my data, I standardize all my features and then generate polynomial features based on ...
1
vote
0answers
154 views

how is countvectorizer used in real production environment?

how is countvectorizer used in real production environment? do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?...
2
votes
2answers
68 views

How to use a dataset where attribute names are changed?

I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed. My goal is to use dataset ...
0
votes
2answers
111 views

Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)

I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following: ...
0
votes
1answer
107 views

Is pre-processing always neccessary?

I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary. I have a lot of ...
1
vote
2answers
740 views

Binary classification: best ways to pre-procees the data

About the dataset I have a training dataset of 129 columns(last column being the classes, i.e., y values) 6068 rows I have to train some algo to do binary classification. The data set has 701 ...
-1
votes
1answer
328 views

Image Classification, Convolution Network and Gamma Correction for images

I am working with NIST Special Database 4, which is a database of fingerprints. The objective is to train a convolutional network (CNN) and train it to classify the fingerprints. After I looked ...
1
vote
0answers
87 views

Transformation of Dependent and Independent Variables

I have a few Independent variables that's normal and a Dependent variables that's skewed , I pick log(feature+SHIFT) to correct skewness. The procedure I follow to get prediction is just take exp(...
0
votes
1answer
93 views

Data preprocessing, relative scale problems in features of same type

I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example. ...