People who code: we want your input. Take the Survey

Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
2
votes
1answer
405 views

Positively skewed target label in regression

I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-...
0
votes
1answer
60 views

How to deal with name strings in large data sets for ML?

My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later. Some word embedding techniques were used for longer text ...
2
votes
1answer
30 views

Is feature scaling at all needed for a feature set with a single feature?

I understand that feature scaling is required to bring features in different magnitudes on a common scale so the model is not biased towards features with higher magnitudes. But if there is only a ...
0
votes
3answers
3k views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
0
votes
2answers
149 views

Standardization with positive and negatives

I have a data set that has a few columns such as: Total cost: mean = 3,000,000 Percent complete: mean = 50 final profit %: mean = 14 I know with such different orders of magnitude before I fit a ...
1
vote
1answer
61 views

Data scaling for large dynamic range in neural networks

The usual strategy in neural networks today is to use min-max scaling to scale the input feature vector from 0 to 1. I want to know if the same principle holds true if our inputs have a large dynamic ...
1
vote
1answer
141 views

Customize loss function for Music Generation LSTM (?)

I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract ...
0
votes
1answer
48 views

How to discretize certain features with a feature set?

I am working with typing data with timing features(unit: ms) and some of the features are based on the keyboard keyCodes(positive integers, range:[8, 222]). Currently, I use ...
1
vote
1answer
55 views

Forecasting with a Machine Learning Algorithm

Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all ...
3
votes
1answer
38 views

preprocessing time sequence

I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event. The preprocessing steps i took are: (1) turning to OneHotEncoding using pandas: <...
1
vote
1answer
20 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
0
votes
1answer
11 views

Find differences of two protein sequence files

I have two separate files A and B of more than 100000 records of protein sequences. Now I need to find the sequences that are in ...
1
vote
1answer
69 views

How to control for Co-variate shift in test data set compared to train data for regression task?

I am working on a regression project. But I am facing the problem of covariate shift in features due to time delay.Test data was collected a year later due to which there has been some change in ...
2
votes
2answers
496 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
1
vote
0answers
17 views

What would be the best way to combine several denoised images and achieve superior denoising results? [closed]

I have generated denoised images using several models and would like to create an ensemble of these individual images to achieve superior denoising results. What would be the best way to combine (...
2
votes
1answer
147 views

Clustering time series based on monotonic similarity

Context I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, ...
1
vote
1answer
6k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
2
votes
1answer
31 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
0
votes
0answers
12 views

The tag preprocessing says it is a data-mining technique ! Is it a correct definition?

This site appears to give Wrong definition of preprocessing of data.Data-mining has no connection with preprocessing of data. The data preprocessing focuses on data preparation for developing software ...
0
votes
2answers
267 views

Error when trying .transform for OrdinalEncoder from Scikit Learn

I'm having a lot of issues using scikit learn recently and was hoping someone could help me with my problem. I can use other methods to ordinal encode but i want to figure this one out. ...
1
vote
2answers
44 views

How to preprocess tensorflow imdb_review dataset [closed]

I am using tensorflow imdb_review dataset, and I want to preprocess it using Tokenizer and pad_sequences When I am using the Tokenizer instance and using the following code: ...
1
vote
0answers
72 views

Sliding Window Normalization

I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize ...
0
votes
0answers
16 views

Pre-processing data - Dataframe manipulation Time Series

I have a question in which I'm not entirely sure in which path to take. I'd appreciate if you could point me in the right direction. Below a screenshot of the a few records of my dataset. As you can ...
0
votes
0answers
31 views

Take several points for each period in a machine learning model

Problem presentation I am working on a prediction model where I must find out if a boat will go back to the same offshore workplace after spending time in a port. When a boat is in a port, it can stay ...
0
votes
0answers
27 views

How do I identify the postulated 600 object classes in OpenImages v. 6?

I have downloaded the OpenImages version 6 image sets (train, validation, test) from CVD Foundation as per instructions from the OpenImages homepage. The OpenImages homepage states that there are 600 ...
0
votes
0answers
12 views

Handling too much range in texts' length

I have a classification problem where inputs are news article and target are topics. However, articles length are very varying from 100 words to 800! I'm using TfIdf and I think that it can puzzle the ...
0
votes
1answer
45 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
1
vote
1answer
153 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
16 views

How to preprocess data for multiple rows instance? [closed]

I have a machine learning problem where I must detect anomalies in (let say) taxes declaration. So I have multiple rows for each enterprise, each row describing the tax declaration a the time (date,...
0
votes
2answers
32 views

Is there any different between feature selection and pca? If there is could anyone please kindly explain for me please?

First of sorry for asking a possibly beginner question, but i don't understand pca seems to be the same as feature selection, but when i search online they seems to be talked differently. What people ...
0
votes
1answer
1k views

How to use inverse_transform in MinMaxScaler for pred answer in a matrix

I am working on a data, for preding output, I used SVR by bellow code: ...
1
vote
0answers
34 views

Variable TimeStep in Spatio Temporal 3DConvNet

I'm working on a new project on climate data. I suppose my output $Y(H*W*1)$ is a consequence of a given data scenario $X(C=4*T*H*W)$ and an initial state $Y0(H*W*1)$. I've chosen a temporal ...
0
votes
0answers
28 views

How to make a Label Encoder trained on a training dataset transform an unseen value of a test dataset?

During the data preprocessing stage, I decided to apply the Label Encoding on one of the columns because it contained data points in string format. Suppose the column contains the following distinct ...
1
vote
1answer
30 views

Python : data type handling by sklearn and impact on memory usage and performance

I am currently working on reducing the memory usage of my data (Pandas DataFrame). This is going quite well : downcasting floats into smaller floats, integers into smaller ones and transforming string ...
0
votes
0answers
29 views

How can I compare imputation techniques on a dataset with sci-kit learn?

I have a dataset data that has missing values. I am trying two ways of imputing these values, but I would like to compare them. In the first method I am using a ...
34
votes
3answers
37k views

StandardScaler before and after splitting data

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
1
vote
1answer
603 views

Is there any NLP library or package which can help in adding comma, punctuation, newlines appropriately to text?

I have a movie transcript without commas, punctuation, or newlines. Is there any NLP technique that can help to implement this?
0
votes
0answers
13 views

Transforming binary data for decision trees

I have binary columns in my dataset (20) e.g. hot_weather, discount (y or no), where in each case 1 = yes no = 0. I am using this data on tree based methods. It is a regression problem and my RMSE is ...
0
votes
1answer
24 views

How to expand abbrevations in text during preprocessing?

Im doing preprocessing on english text data. I have some domain specific abbreviations, for which i'm maintaining internal dictionary with key-value pairs. The problem i'm facing is the text has ...
0
votes
1answer
39 views

Normalization for a 2d input array

I am new to machine learning and trying to apply it to my problem. I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there ...
0
votes
0answers
7 views

Factors in choosing a continuous encoder?

Apart from heuristically treating an encoder like a hyperparameter, how can one decide which encoder to use for continuous features? Rule out encoders that do/ don't accept negative values. Can we ...
1
vote
0answers
28 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
1
vote
1answer
46 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...
1
vote
1answer
19 views

For obejct detection, should I resize my custom images first and then start the annotation or it won't matter?

I have my custom dataset images of size (1080 x 1920) and I am trying to use yolov3 for object detection. I noticed that yolov3 model accepts an input image size of 416 x 416. So I am in confusion if ...
2
votes
1answer
252 views

What is the best way to deal with image sizes in object detection using YOLO v. 4?

I am using YOLO v. 4 to perform object detection and classification on a large number of images. I do this through the Python interface. My images are all sorts of sizes and aspect ratios. Examples of ...
1
vote
1answer
57 views

Remove correlated features before or after splitting test and training set?

I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it ...
0
votes
1answer
66 views

can you suggest best method for background / ambient noise removal from cough audio and keep only cough signal

I'm performing background/ambient noise removal form cough signal and keep only cough audio signals using python.Can you suggest best noise filters to achieve this and keep only cough audio signal ...
0
votes
0answers
73 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
2
votes
1answer
1k views

Training a LSTM on a time serie containing multiple inputs for each timestep

I am trying to train a LSTM in order to use it for forecasting. The problem is basically a multivariate multi-step time series problem. It is simply an experiment to see how statistical models (ARIMA, ...

1
2
3 4 5
8