Questions tagged [preprocessing]
Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.
541
questions
-1
votes
0
answers
8
views
Unorthodox Time Series Preprocessing methods
I am writing a school paper on preprocessing of time series. In the paper I want to test the unorthodox methods of time series preprocessing. I want to then feed the data to LSTM model and measure its ...
0
votes
0
answers
43
views
Identifying patterns and trends in insurance behavior with clustering or decision trees
I have a list of patient accounts based on their discharge date. I have various inputs related to each patient such as their financial class, insurance information, demographics, claims information, ...
0
votes
1
answer
8
views
Faster preprocessing for Arabic texts
Background
I'm analyzing a relatively large text-based Arabic dataset using Python (50,000 - 70,000 text files; total size ~5GB).
I want to segment, stem, and POS tag the dataset. I am aware of two ...
- 1
0
votes
0
answers
9
views
Index construction for multivariate ordinal with known weights
Constructing a prioritization score from the following ordinal variables:
...
0
votes
0
answers
11
views
What is the intuition behind designing a neural network for complex non-linear regression problems?
What is the intuition behind designing a neural network for complex non-linear regression problems?
I'm looking for guidance on how to intuitively design a neural network for regression, specifically ...
0
votes
0
answers
43
views
How does one handle a dataset with groups of features and groups of labels in classification?
I have a large dataset (1.8mil samples). There are 15 features: x1, y1, z1, e1, d1, x2,..., d3. (x,y,z) are coordinates, e is energy, and d is a derived feature- Euclidean distance between the ...
- 1
0
votes
0
answers
30
views
Mobilenetv2 transfer learning
Goal:
Transfer learning Mobilenetv2 (input size 224x224 and it's own preprocessing (resize + central_crop + normalization)) as encoder for Unet with input size 512x512 using pytorch.
What I've done:
...
0
votes
1
answer
24
views
How to preprocess/encode categorical data, to use in dimensionality reduction and clustering algorithms?
I am working on a project witht the goal of clustering participants of in a survey according to their answers. The dataset is a set of 63 questions, some nominal and some ordinal. How should I encode ...
0
votes
0
answers
26
views
How to choose thresholds to discretize target for binary classification
My group is using logistic regression to investigate the most predictive features in a dataset. Our target variable is actually a continuous variable that we discretized using two cutoff thresholds (...
0
votes
0
answers
6
views
Classification for multi row observation: Long format to Wide format always efficient?
I have a table of observations, or rather 'grouped' observations (that spans more than a row), where each group represents a deal, and each row representing a product. But the prediction is to be done ...
- 143
0
votes
0
answers
10
views
Resampling calibration data
Is it appropriate to resample a calibration dataset, or should the class distribution be consistent with the observed data?
- 1
0
votes
1
answer
45
views
Why does forecasting with an LSTM yield better results with shuffling?
I first partition the timeseries data into train, validation, and test splits, without performing any shuffling.
Each row is a window of ordered samples, so my training data might be shaped ...
- 706
0
votes
0
answers
21
views
How to train Logistic regression model with multiple inputs for 1 target value?
My data looks like similar to this: (the picture below is not mine, but describes perfectly my situation)
where the IDs are not unique but for each ID value I have a unique target value
The following ...
0
votes
0
answers
11
views
Help required in opening files of a dataset (.phys, .thermal, .pts, .ass extensions)
We have received a dataset that consists of audio, visual, thermal, and physiological modalities. Upon exploring the dataset, we encountered some challenges in opening the following file types:
.phys ...
- 101
0
votes
0
answers
19
views
Outlier Detection in Zero-inflated dataset
I have a feature , more than 85% of its values are 0s. So when i try traditional boxplot method for outlier detection, all the non-zero values are identified as outliers. What modifications can i make ...
0
votes
0
answers
15
views
Tips for pre-processing long unstructured text?
I'm working on a React-Node project where I'm trying to pre-process some text before passing it to GPT-4 to perform information extraction out of it, the flow of the project is:
User uploads document
...
1
vote
1
answer
73
views
Reducing emails token count preprocessing for Large Email Datasets - Feeding LLMs
I have a large email dataset in .txt format and want to feed LLMs (like Gemini and ChatGPT) to provide answers based on email content.
The token count for my email data is very high (~1M for 1K emails)...
- 113
1
vote
0
answers
24
views
1
vote
1
answer
61
views
Treat Age as Categorical or Continuous
I am in the middle of processing data for feature reduction (haven't decided the method yet). The data consists of a row of people and ~3000 column of attributes that correspond to that person and is ...
- 11
0
votes
0
answers
29
views
Outliers in test data
This famous dataset have a lot of zero values (Above 500). But it is not really clear, some of them are outliers or not.
My question is: What if i decided that ALL 0's are outliers (it's pretty likely)...
- 43
1
vote
0
answers
51
views
How do I know the appropriate number of iterations when using Miceforest for imputation?
I want to know how to avoid overfitting without having to increase the number of iterations excessively in Python with the Miceforest library. I know you can make a correlation map of data sets but I ...
0
votes
1
answer
16
views
References for relative/scaled time?
I have time-based (e.g., patient) data where some subjects (patients) were treated for a longer time than others. I want to analyze the treatment effects where observations are really noisy (so ...
- 352
0
votes
0
answers
8
views
Strategies for Encoding Large Datasets in Symbolic Music Generation for BERT-type Model
I am creating a BERT-type model for symbolic music generation. An observation of my database is a musical piece. Actually, is a "viewpoint" of the piece: ...
- 101
0
votes
0
answers
12
views
Dataset with replicates: aggregation or not?
I am currently developing a neural network tailored to a regression problem using a synthetic dataset derived from an experimental simulation campaign. Given the stochastic nature of both the ...
0
votes
0
answers
79
views
Heavy right skewed target
I scrapped Data science job postings of almost 1000 jobs for salary prediction. I wanted to bin the salary column and build a classification model from the job descriptions. But the (salary)target is ...
- 21
0
votes
0
answers
39
views
What feature selection method is ideal for a large dimensional data frame after the result of one hot encoding?
I am trying to solve a sports related multi class classification problem in Python, I aim to train a custom neural network and also a SVM. I have performed prior data cleaning and encoded my data ...
0
votes
1
answer
27
views
Academic name of dataset preparation method with hierarchical-learned labels? - E.g., cold→half-cooked→cooked
What's the name of the dataset preparation method indicating hierarchical ontologies?
Assume photos of cold, half-cooked, and fully-cooked chickens. Annotate with temperature data. E.g., at current ...
- 101
0
votes
0
answers
27
views
Alternatives to Model-Based Feature Selection for Unsupervised Clustering
I am running a clustering model on a group of patients who are hypertensive with hopes of identifying different variations in clinical characteristics among hypertensive individuals.
One of the issues ...
0
votes
0
answers
23
views
how to give numeric value for a categorical feature instances
i have a data set with day of the week. i want to transform this to numerical in the data preprocessing stage. how can i do this? I want to do this in Orange data mining software...thank you
0
votes
1
answer
76
views
Correct order of preprocessing/EDA/feature engineering?
I was wondering if I have the correct order of preprocessing/EDA/feature engineering below?
Yes there are nuances and may vary from problem to problem, but am just looking for a general pipeline for ...
- 111
0
votes
0
answers
56
views
Mislabeled problem with hospital data
I am writing this post to ask or see if someone can help me with this problem in case you may have faced a similar situation.
My problem has to do with a ML tool that I am trying to develop in a ...
- 1
0
votes
0
answers
25
views
How to infer the following graphs for dimensionality reduction?
I'm dealing with a high-dimensional(1600 features, 9500 columns) binary classification problem. My current accuracy, and other metrics are not upto the mark. I am trying different feature selection ...
0
votes
0
answers
54
views
Outliers problem
everyone hope you all are good.
I have a hard time with outliers
First i found some outliers in train set then i capped them and did the same with test set. So when i checked my rsme root mean ...
0
votes
1
answer
63
views
Time Series Forecasting for Multiple Store Sales with Simultaneous Timestamps
I have a sales dataset with each store having a unique identifier. The dataset contains daily sales data for each store over a period of around two-years. I'm looking to build a time series ...
0
votes
0
answers
25
views
Data Preprocessing in the Wild?
I am new to ML, NN, and data science as a whole so the following question might sound silly. How can we perform inference when the model is deployed in the wild?
To my understanding, cleaning/...
0
votes
0
answers
25
views
Preprocessing overheads in Machine Learning
Meta reports that data preprocessing overheads is fast becoming a bottleneck to machine learning training (https://engineering.fb.com/2022/09/19/ml-applications/data-ingestion-machine-learning-...
0
votes
0
answers
38
views
Handling Missing Values
I have the following artificial data :
Now I am trying to handle missing values in the age and salary columns using mean imputation. I am using the following code to do so :
...
- 231
0
votes
0
answers
13
views
How to train model with data received from different 3x accelerometers sensors?
I want to make a model based on accelerometer data to recognise different activities like running, walking etc. I have a small dataset collected from my target sensor. I found another dataset with ...
- 101
0
votes
0
answers
8
views
Grouped Time Restricted Demand Regression with value cap
so I am working on quite an interesting regression task that I haven't encountered before.
Our company sells products (steel) in tons. We offer contracts where the customer orders a certain amount of ...
- 119
1
vote
1
answer
373
views
What is the best way to avoid data leaking in timeseries forecasting multiple labels?
I'm predicting crypto pairs volume. I want to increase my accuracy by using one model for different pairs. Question is how to avoid timedata leaking?
Example n - pairs, t- time, m - features for each. ...
- 11
0
votes
1
answer
159
views
different range of target values in neural network
I am working on a neural network regression code. The dataset includes 14 features in the range value between -1 and 1. while the target variable is changing among (0.000759) to (1100).
The target ...
- 1
1
vote
3
answers
2k
views
Recommended way to embed a text thousands of tokens long?
I've split the text up sections each 512 tokens long and created embeddings for each of them.
I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?
...
- 288
1
vote
1
answer
552
views
How to handle date columns in a dataset with over 100+ date columns?
Background: I have a dataset with around 85k rows and 320 columns.I have no formal domain knowledge and the columns ain't intuitive as well the dataset is not in a language that I speak or understand. ...
0
votes
3
answers
40
views
Data preprocessing
I just want to know how to determine whether to remove the missing values or to impute them with mean , median or mode. I usually remove the missing values but it decreases the size of the dataset by ...
0
votes
2
answers
45
views
Scaling of categorical feautres
In the context of algorithms that consider the scales of features, I have a situation where some features are encoded using ordinal encoding, some features are binary, and some features are standard ...
- 51
0
votes
1
answer
426
views
Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value
I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
- 1
0
votes
0
answers
291
views
Can I convert a TensorFlow tensor to a CSV file?
I'm working on a semantic segmentation problem. I have saved my images in a tensor of shape (4767, 192, 192, 3) [It contains 4767 images of size 192192 with RGB channels]. On the other hand, I have ...
- 314
0
votes
0
answers
38
views
How to predict continous values using resnet regression model
I'm making a resnet regression model to predict force plate data using smartinsole data. so my data input is smartinsole data and the force plate is the label. below is an example of my data sample.
<...
- 31
1
vote
1
answer
215
views
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and no other ...
- 113
1
vote
0
answers
45
views
Labelling spectrograms
Currently I'm working on a ML project, just need an information, is there any tool that is present that can load audios file and generates spectrograms as well as an option to annotating/ label the ...
