Skip to main content

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

How can i retrieve incorrect labelled predictions on new unseen data on image classification when i want to retrain my model later on?

I've recently made a binary image classification model with transfer learning. The model is used on an api and the predictions gets saved into a database. The problem is that it predicts images ...
0 votes
0 answers
10 views

Tips for pre-processing long unstructured text?

I'm working on a React-Node project where I'm trying to pre-process some text before passing it to GPT-4 to perform information extraction out of it, the flow of the project is: User uploads document ...
0 votes
1 answer
209 views

How to handle ID variables when splitting data for machine learning?

I'm new to machine learning, and I'm working with some international head-to-head sports competition data. I used relational data creation techniques in tidyverse to join several data sources to ...
1 vote
1 answer
138 views

Grouping profiles strings having the same words, but occurring out of order

I have a dataframe containing a column of profile types, which looks like this: ...
0 votes
0 answers
5 views

Variational AutoEncoder using a matrix as dataset with a time component or ...?

The project is to repair data/smoothing. I would like to use a matrix of shape (5,13,3922): 5 differents types 13 features, one of them is the Times series, dates in format yyyy-mm-dd that I turned ...
1 vote
1 answer
553 views

Group_by 2 variables and pivot_wider distribution based on 2 others

Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all ...
16 votes
3 answers
15k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
5 votes
4 answers
5k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
0 votes
1 answer
57 views

Do you use categorical data types?

Personally I've never used the categorical data type in pandas, and leave eveything as objects. I've seen it has the capability to be saved as parquet files, saves data etc... What are the pros and ...
1 vote
1 answer
88 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
0 votes
1 answer
715 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
1 vote
1 answer
21 views

Reducing emails token count preprocessing for Large Email Datasets - Feeding LLMs

I have a large email dataset in .txt format and want to feed LLMs (like Gemini and ChatGPT) to provide answers based on email content. The token count for my email data is very high (~1M for 1K emails)...
1 vote
1 answer
181 views

One Hot Encoding where all sequences don't have all values

Is there a way (other than manually creating dictionaries) to one hot encode sequences in which not all values can be present in a sequence? sklearn's OneHotEncoder ...
0 votes
1 answer
15 views

Missing data in train set and test set

I have a dataset of N columns. Now I'm able to preprocess data and find a subset of features that I can use to train a model and make predictions. In the case where the train data has missing feature ...
0 votes
1 answer
24 views

Combining datasets with different featues but, have similar names but, not exactly I can do name = name

I want to join datasets of data I have scraped from Open Critic, vgchartz, and Steam. All of these datasets have different fields but, I want to be able to join the 3 datasets to make one dataset with ...
0 votes
0 answers
8 views

Missing Value in a dataset

i'm curently cleaning a dataset and theres a column called " Institution", it is needed for encoding and training the classification model later so it needs to be cleaned. In that column, ...
0 votes
0 answers
5 views

What's the intuitively best option for preprocessing images of varying aspect ratios and resolutions?

We've got a complex pipeline that produces a crop of the minimum possible subset image that includes all useful information. We've got a number of customers using various models on these images, and ...
0 votes
0 answers
14 views

I want to import my states legislation into a database. I need advice in parsing large HTML, TXT into an importable form. It is Large PDF initialy

My data consists of large pdf files containing state legislation. I can convert the pdf to xml, txt or html, and I can manually go through, identify the tables and write a script to extract only the ...
1 vote
1 answer
307 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
0 votes
0 answers
10 views

what are Non-AI methods in detecting how much brown pixels an image have

looking for a non-AI approach on how to find how much "brown" color a 256x256 px image have. I am making a dataset that will be used as training data for AI later on. There are several ...
1 vote
1 answer
27 views

How to match the two datasets on Electoral bonds given by SBI?

Just stumbled across the Electoral bond dataset https://www.kaggle.com/datasets/shaundanielll/electoral-bond-data-state-bank-of-india There are two csvs- One contains Encashment data(Date of ...
0 votes
2 answers
135 views

How to convert city name prefix abbreviations?

Is there any standard tool or library or list for expanding town name abbreviations? For example "MT HOLLY" -> "MOUNT HOLLY" or "ST MICHAELS" -> "SAINT ...
1 vote
1 answer
21 views

Treat Age as Categorical or Continuous

I am in the middle of processing data for feature reduction (haven't decided the method yet). The data consists of a row of people and ~3000 column of attributes that correspond to that person and is ...
1 vote
1 answer
277 views

Splitting Temporal Data

I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in ...
1 vote
2 answers
133 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
0 votes
1 answer
97 views

Multivariate data preprocessing

I am trying to understand how multivariate data preprocessing works but there are some questions in my mind. For example, I can do data smoothing, transformation (box-cox, differentiation), noise ...
1 vote
2 answers
3k views

Pandas: To find duplicate columns

I have a dataframe with different dtypes like int, float, object, datatime etc. I am performing data cleaning, to list or find ...
1 vote
2 answers
6k views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
0 votes
1 answer
53 views

Outlier Handing when most value is 0

Just a question, i know that when we plot against the distribution of numerical data, those who fall outside of the boxplot (diamond shape point) are considered outlier. However, i met a case where ...
1 vote
1 answer
120 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
0 votes
1 answer
32 views

How can we make embeddings from the text?

I have a dataset with questions and answers to them. I want to make embeddings of questions and save them in a vector database. Next, I will make a query to the database. With the help of the pinecone ...
0 votes
2 answers
316 views

Missing Values Correlation

Is it worth it to study missingness correlation between columns? If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at ...
0 votes
1 answer
142 views

Is there a standardized way to do data analysis?

Is there a standard way to do data analysis? So for example, something like this: 1. Data mining 2. Data cleaning 3. xx 4. Data and result interpretations I ...
0 votes
1 answer
186 views

How can I calculate total days past due between billing events?

I am dealing with a dataframe with subscription events partitioned by username, subscription status, and relative timestamps. For each of the dates, there are changes in time when the subscription ...
1 vote
2 answers
634 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
0 votes
1 answer
127 views

Whether to replace NaN values in result column

I have a training dataset where we have to predict "Result" based on features "A", "B", "C" and "D" using machine learning. For a few rows, the "...
0 votes
1 answer
424 views

What's the best approach to dealing with missing data in a dataset?

I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
0 votes
2 answers
864 views

What is the best solution to replace NaN values?

I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that ...
0 votes
0 answers
45 views

Heavy right skewed target

I scrapped Data science job postings of almost 1000 jobs for salary prediction. I wanted to bin the salary column and build a classification model from the job descriptions. But the (salary)target is ...
0 votes
1 answer
269 views

Group a spark dataframe by a starting event to an ending event

Given a series of events (with datetime) such as: failed, failed, passed, failed, passed, passed I want to retrieve the time from when it first "failed" to when it first "passed," ...
1 vote
2 answers
2k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
0 votes
0 answers
15 views

How to identiyfy issues in structure or missing values in netCDF-file that contains weather data?

during processing of weather data files (wind x,y 10m), i get the follwing error: height3' not present in all datasets and coords='different'. Either add 'height3' to datasets where it is missing or ...
0 votes
0 answers
15 views

How to generate a netCDF weather forecast for a missing day based on yesterdays forecast?

as can be seen here met.no, the forcast is missing for 20210823. This forcast for each day is a 66 hour forecast, mening it will cover the next "missing" day. Not knowing much about netCDF ...
0 votes
1 answer
910 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
2 votes
2 answers
240 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
1 vote
1 answer
727 views

Downsampling audio files for use in Machine Learning

I'm trying to use the work (Neural Networks) done in this repo: https://github.com/jtkim-kaist/VAD It says this: Note: To apply this toolkit to other speech data, the speech data should be ...
0 votes
0 answers
16 views

Iterating more quickly, shorter feedback cycles, general workflow practices

(caveat: I've not written that much ML code yet) When I'm writing non-ML code I like to create short feedback loops to check the validity of my code. I use unit-tests or sometimes short helper tools (...
1 vote
3 answers
5k views

How to sort a multi-level pandas data-frame by a particular column?

I would like to sort a multi-index pandas dataframe by a column, but do not want the entire dataframe to be sorted at once. But rather would like to sort by one of the indices. Here is an example of ...
1 vote
2 answers
1k views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
1 vote
3 answers
698 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...

1
2 3 4 5
16