Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Outliers in test data

This famous dataset have a lot of zero values (Above 500). But it is not really clear, some of them are outliers or not. My question is: What if i decided that ALL 0's are outliers (it's pretty likely)...
Тима 's user avatar
1 vote
1 answer
29 views

How do I know the appropriate number of iterations when using Miceforest for imputation?

I want to know how to avoid overfitting without having to increase the number of iterations excessively in Python with the Miceforest library. I know you can make a correlation map of data sets but I ...
Eduardo Dimas's user avatar
0 votes
1 answer
14 views

References for relative/scaled time?

I have time-based (e.g., patient) data where some subjects (patients) were treated for a longer time than others. I want to analyze the treatment effects where observations are really noisy (so ...
sds's user avatar
  • 352
0 votes
0 answers
6 views

Strategies for Encoding Large Datasets in Symbolic Music Generation for BERT-type Model

I am creating a BERT-type model for symbolic music generation. An observation of my database is a musical piece. Actually, is a "viewpoint" of the piece: ...
Kikolo's user avatar
  • 101
0 votes
0 answers
10 views

Dataset with replicates: aggregation or not?

I am currently developing a neural network tailored to a regression problem using a synthetic dataset derived from an experimental simulation campaign. Given the stochastic nature of both the ...
Andrea Ferrari's user avatar
0 votes
0 answers
41 views

Heavy right skewed target

I scrapped Data science job postings of almost 1000 jobs for salary prediction. I wanted to bin the salary column and build a classification model from the job descriptions. But the (salary)target is ...
Sendhan's user avatar
  • 21
0 votes
0 answers
35 views

What feature selection method is ideal for a large dimensional data frame after the result of one hot encoding?

I am trying to solve a sports related multi class classification problem in Python, I aim to train a custom neural network and also a SVM. I have performed prior data cleaning and encoded my data ...
pastybake2002's user avatar
0 votes
1 answer
26 views

Academic name of dataset preparation method with hierarchical-learned labels? - E.g., cold→half-cooked→cooked

What's the name of the dataset preparation method indicating hierarchical ontologies? Assume photos of cold, half-cooked, and fully-cooked chickens. Annotate with temperature data. E.g., at current ...
Samuel Marks's user avatar
0 votes
0 answers
20 views

Alternatives to Model-Based Feature Selection for Unsupervised Clustering

I am running a clustering model on a group of patients who are hypertensive with hopes of identifying different variations in clinical characteristics among hypertensive individuals. One of the issues ...
Zory Dory's user avatar
0 votes
0 answers
17 views

how to give numeric value for a categorical feature instances

i have a data set with day of the week. i want to transform this to numerical in the data preprocessing stage. how can i do this? I want to do this in Orange data mining software...thank you
Jerry Campbell's user avatar
0 votes
1 answer
51 views

Correct order of preprocessing/EDA/feature engineering?

I was wondering if I have the correct order of preprocessing/EDA/feature engineering below? Yes there are nuances and may vary from problem to problem, but am just looking for a general pipeline for ...
Katsu's user avatar
  • 111
0 votes
0 answers
44 views

Mislabeled problem with hospital data

I am writing this post to ask or see if someone can help me with this problem in case you may have faced a similar situation. My problem has to do with a ML tool that I am trying to develop in a ...
Daniel's user avatar
  • 1
0 votes
0 answers
24 views

How to infer the following graphs for dimensionality reduction?

I'm dealing with a high-dimensional(1600 features, 9500 columns) binary classification problem. My current accuracy, and other metrics are not upto the mark. I am trying different feature selection ...
Tanmay Sharma's user avatar
0 votes
0 answers
50 views

Outliers problem

everyone hope you all are good. I have a hard time with outliers First i found some outliers in train set then i capped them and did the same with test set. So when i checked my rsme root mean ...
Shoaib Ahmed's user avatar
0 votes
1 answer
38 views

Time Series Forecasting for Multiple Store Sales with Simultaneous Timestamps

I have a sales dataset with each store having a unique identifier. The dataset contains daily sales data for each store over a period of around two-years. I'm looking to build a time series ...
SIDDHARTH CHAVAN's user avatar
0 votes
0 answers
24 views

Data Preprocessing in the Wild?

I am new to ML, NN, and data science as a whole so the following question might sound silly. How can we perform inference when the model is deployed in the wild? To my understanding, cleaning/...
Abdullah Abdulrahman's user avatar
0 votes
0 answers
21 views

Preprocessing overheads in Machine Learning

Meta reports that data preprocessing overheads is fast becoming a bottleneck to machine learning training (https://engineering.fb.com/2022/09/19/ml-applications/data-ingestion-machine-learning-...
Rajath Shashidhara's user avatar
0 votes
0 answers
32 views

Handling Missing Values

I have the following artificial data : Now I am trying to handle missing values in the age and salary columns using mean imputation. I am using the following code to do so : ...
John adams's user avatar
0 votes
0 answers
13 views

How to train model with data received from different 3x accelerometers sensors?

I want to make a model based on accelerometer data to recognise different activities like running, walking etc. I have a small dataset collected from my target sensor. I found another dataset with ...
eugenn's user avatar
  • 101
0 votes
0 answers
8 views

Grouped Time Restricted Demand Regression with value cap

so I am working on quite an interesting regression task that I haven't encountered before. Our company sells products (steel) in tons. We offer contracts where the customer orders a certain amount of ...
Martin Pichler's user avatar
0 votes
0 answers
22 views

Image classification of centered objects with convolutional neural networks

Given that I have a set of images that contain multiple objects for which labels exist and the object the image label refers to is always in the center. The objects vary in size. I want to train a ...
fhllw's user avatar
  • 1
1 vote
1 answer
145 views

What is the best way to avoid data leaking in timeseries forecasting multiple labels?

I'm predicting crypto pairs volume. I want to increase my accuracy by using one model for different pairs. Question is how to avoid timedata leaking? Example n - pairs, t- time, m - features for each. ...
Dima's user avatar
  • 11
0 votes
1 answer
72 views

different range of target values in neural network

I am working on a neural network regression code. The dataset includes 14 features in the range value between -1 and 1. while the target variable is changing among (0.000759) to (1100). The target ...
Mali's user avatar
  • 1
1 vote
2 answers
959 views

Recommended way to embed a text thousands of tokens long?

I've split the text up sections each 512 tokens long and created embeddings for each of them. I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended? ...
codeananda's user avatar
0 votes
1 answer
215 views

How to handle date columns in a dataset with over 100+ date columns?

Background: I have a dataset with around 85k rows and 320 columns.I have no formal domain knowledge and the columns ain't intuitive as well the dataset is not in a language that I speak or understand. ...
Krish Athreyam's user avatar
0 votes
3 answers
33 views

Data preprocessing

I just want to know how to determine whether to remove the missing values or to impute them with mean , median or mode. I usually remove the missing values but it decreases the size of the dataset by ...
Yousaf Chaudhary's user avatar
0 votes
0 answers
15 views

TensorFlow lite classification model;{1,1,1,3}

How can I initialize and insert images into this classification model; what is meant by {1,1,1,3} according to my parameters it should be {1,size,size,3} please help try { Firstclass model = ...
Mohammad Nabeel's user avatar
0 votes
2 answers
40 views

Scaling of categorical feautres

In the context of algorithms that consider the scales of features, I have a situation where some features are encoded using ordinal encoding, some features are binary, and some features are standard ...
jxqbbb's user avatar
  • 51
0 votes
1 answer
134 views

Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value

I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
vito97's user avatar
  • 1
0 votes
0 answers
185 views

Can I convert a TensorFlow tensor to a CSV file?

I'm working on a semantic segmentation problem. I have saved my images in a tensor of shape (4767, 192, 192, 3) [It contains 4767 images of size 192192 with RGB channels]. On the other hand, I have ...
PicaR's user avatar
  • 314
0 votes
0 answers
31 views

How to predict continous values using resnet regression model

I'm making a resnet regression model to predict force plate data using smartinsole data. so my data input is smartinsole data and the force plate is the label. below is an example of my data sample. <...
stack offer's user avatar
1 vote
1 answer
100 views

Which preprocessing is the correct way to forecast time-series data using LSTM?

I just started to study time-series forecasting using RNN. I have a few months of time series data that was an hour unit. The data is a kind of percentage value of my little experiment and no other ...
orde.r's user avatar
  • 113
0 votes
0 answers
12 views

Transform monthly data to the weekly data based on categorical value

I have monthly data like this which is hours: month,hours 1,20 2,19.5 3,21 ... I have weekly data like this which is quantities of the asset: ...
Muhammad Ikhwan Perwira's user avatar
1 vote
0 answers
32 views

Labelling spectrograms

Currently I'm working on a ML project, just need an information, is there any tool that is present that can load audios file and generates spectrograms as well as an option to annotating/ label the ...
Karthik Sudapelli's user avatar
0 votes
0 answers
22 views

Problem with the size of test data?

I am working on my thesis, which has two different research questions: Evaluate transformer models while incorporating non textuel features Evaluate the importance of data quality in tranformer ...
Aymen Ba's user avatar
0 votes
0 answers
20 views

Preparing LSTM Network input from multiple files

I would like to train LSTM Network, which should take 5 files as input and predict the 6th file. Each file contains 810000 data points (precipitation values), and each data point indices the location. ...
karnati shiva's user avatar
2 votes
1 answer
86 views

How to improve the preservation of the global data structure in UMAP?

I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
pmu2022's user avatar
  • 121
0 votes
1 answer
65 views

Keras MLP not working

For my degree's final project I am working with Keras and trying to build different AI models. I'm having trouble with an MLP. First I preprocess the UNSW-NB15 dataset and later use it as input in a ...
alex martinez's user avatar
0 votes
0 answers
15 views

Data processing - how to input a pandas column that contains numbers and numpy arrays

I have a pandas df that contains numbers and strings. I use word2vec to convert all the strings into embeddings. The problem now is that these embeddings are all numpy arrays. So now my pandas df ...
cyc's user avatar
  • 1
0 votes
0 answers
59 views

Why is the shape of my transformed data from CountVectorizer 2x2?

I am currently working on a text classification problem using Python and scikit-learn. I am using CountVectorizer to convert my text data into numerical vectors. I have also applied a preprocessor to ...
Anantashayana Hegde's user avatar
0 votes
0 answers
16 views

Performing imputation only on the test set?

I'm working on a medical machine learning problem. The key challenge is working with small datasets with quite a lot of missing data. Experimentally, I've seen complete-case analysis (i.e. dropping ...
Ben Consterdine's user avatar
0 votes
0 answers
16 views

Best way to regrid data (polar stereographic to regular)

I have data that's in 'polar stereographic' form and I want to regrid it to a regular grid that matches a grid I currently have. I've seen a few example that tend to be fairly complicated. I'd ideally ...
Socorro's user avatar
  • 101
1 vote
1 answer
111 views

How to encode & scale IP addresses as input for ML models

Im currently working on an anomaly detection while making a transaction. As a part of the data that I extracted, I have the IP addresses of the indivduals who made the transaction. Since the IP ...
Sivadithiyan official's user avatar
0 votes
2 answers
78 views

Fine-tune GPT on sketch data (stroke-3)

These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it. I was planning on using some pretrained models that are ...
ilved17's user avatar
  • 31
1 vote
1 answer
47 views

Order of preproccesing, avoiding leakage and metrics

I have a dataset with ~40k records and 16 columns (including the target) and I want to understand the correct process behind whole data science proccess. This is what I did: Performed an EDA which ...
pustelnikk's user avatar
0 votes
0 answers
16 views

Why not decrease the number of filters in each 1-dimensional convolutional layer for spectrogram processing?

In convolutional neural networks, the number of filters usually increases with every convolutional layer. Why is this common practice? Nonetheless, these networks typically process images. When ...
Value_Investor's user avatar
0 votes
1 answer
37 views

Is there a standard order of operations for data preparation?

In what order should I do the following given a dataset: (E)ncoding of Categorical Variables (N)ormalization (B)alancing of data (I)mputation of Missing Values (R)emoval of Duplicates/Infinity/...
Tariq's user avatar
  • 15
4 votes
1 answer
478 views

How can someone build a dataset for a "propensity to purchase" model?

Ok, this might seem a trivial question for some and it's not even a question, more like a discussion. I read the rules and I believe it's everything fine, so I'm gonna take my chances... Here's the ...
Andrew Joplh's user avatar
1 vote
0 answers
111 views

Select rows in Orange based on column value comparison

I'm trying to remove some rows from a cardio dataset where the value for the systolic pressure (ap_hi) is below the diastolic pressure (ap_low) which to me indicates a data error. The code I would ...
Newbie Oranger's user avatar
0 votes
0 answers
15 views

What is the best way to whittle down rendundant categorical data?

I'm trying to build a linear regression model in Tensorflow (and preprocessing with pandas) that will help me categorize bank transactions. I'm trying to whittle down the vendor parameter, because the ...
Ethan Leyden's user avatar

1
2 3 4 5
11