Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

How to build a recommendation system Based on user infos and without ratings?

I would like to build a recommendation system based only on user informations(age,sex,zipcode,and some quiz answers),based on those features i want to recommend assurance products, but i am confused ...
khadija's user avatar
0 votes
0 answers
13 views

What to include in fact and dimension table from election database

I am working with election dataset of India of year 2014 and data for 2019 I also have table for party names and descriptions and finally state name with code. I am not getting how do i create a ...
Sugam Sharma's user avatar
1 vote
1 answer
78 views

Optimized input data structure for ML model training

I have a large dataset (20M+ rows) of user interactions which I want to use to predict the probability of a customer purchasing an item in one-, three- and six months time. However since the ...
MJ_VdH's user avatar
  • 11
0 votes
1 answer
26 views

CS undergrad query about DS

why is learning DS so ambigious .you dont truly know what should you learn to actually do DS .web dev say has a clear path learn html css js and you can make something .i am a cs undergrad just want ...
Muhammad Umer's user avatar
0 votes
0 answers
18 views

How can i retrieve incorrect labelled predictions on new unseen data on image classification when i want to retrain my model later on?

I've recently made a binary image classification model with transfer learning. The model is used on an api and the predictions gets saved into a database. The problem is that it predicts images ...
Enes Aygun's user avatar
0 votes
0 answers
11 views

Tips for pre-processing long unstructured text?

I'm working on a React-Node project where I'm trying to pre-process some text before passing it to GPT-4 to perform information extraction out of it, the flow of the project is: User uploads document ...
Abdul Munem's user avatar
0 votes
0 answers
5 views

Variational AutoEncoder using a matrix as dataset with a time component or ...?

The project is to repair data/smoothing. I would like to use a matrix of shape (5,13,3922): 5 differents types 13 features, one of them is the Times series, dates in format yyyy-mm-dd that I turned ...
Adurrow's user avatar
1 vote
1 answer
34 views

Reducing emails token count preprocessing for Large Email Datasets - Feeding LLMs

I have a large email dataset in .txt format and want to feed LLMs (like Gemini and ChatGPT) to provide answers based on email content. The token count for my email data is very high (~1M for 1K emails)...
Rafael Borja's user avatar
0 votes
1 answer
19 views

Missing data in train set and test set

I have a dataset of N columns. Now I'm able to preprocess data and find a subset of features that I can use to train a model and make predictions. In the case where the train data has missing feature ...
0-0's user avatar
  • 1
0 votes
0 answers
10 views

Missing Value in a dataset

i'm curently cleaning a dataset and theres a column called " Institution", it is needed for encoding and training the classification model later so it needs to be cleaned. In that column, ...
Harry lou's user avatar
0 votes
0 answers
6 views

What's the intuitively best option for preprocessing images of varying aspect ratios and resolutions?

We've got a complex pipeline that produces a crop of the minimum possible subset image that includes all useful information. We've got a number of customers using various models on these images, and ...
Tal's user avatar
  • 101
0 votes
0 answers
10 views

what are Non-AI methods in detecting how much brown pixels an image have

looking for a non-AI approach on how to find how much "brown" color a 256x256 px image have. I am making a dataset that will be used as training data for AI later on. There are several ...
DrakeJest's user avatar
  • 101
1 vote
1 answer
27 views

How to match the two datasets on Electoral bonds given by SBI?

Just stumbled across the Electoral bond dataset https://www.kaggle.com/datasets/shaundanielll/electoral-bond-data-state-bank-of-india There are two csvs- One contains Encashment data(Date of ...
dirtynerdy's user avatar
1 vote
1 answer
25 views

Treat Age as Categorical or Continuous

I am in the middle of processing data for feature reduction (haven't decided the method yet). The data consists of a row of people and ~3000 column of attributes that correspond to that person and is ...
sheroam's user avatar
  • 11
0 votes
1 answer
29 views

Combining datasets with different featues but, have similar names but, not exactly I can do name = name

I want to join datasets of data I have scraped from Open Critic, vgchartz, and Steam. All of these datasets have different fields but, I want to be able to join the 3 datasets to make one dataset with ...
Christopher Kinoshita's user avatar
0 votes
1 answer
143 views

Outlier Handing when most value is 0

Just a question, i know that when we plot against the distribution of numerical data, those who fall outside of the boxplot (diamond shape point) are considered outlier. However, i met a case where ...
Razark's user avatar
  • 1
0 votes
1 answer
33 views

How can we make embeddings from the text?

I have a dataset with questions and answers to them. I want to make embeddings of questions and save them in a vector database. Next, I will make a query to the database. With the help of the pinecone ...
7wafer7's user avatar
0 votes
0 answers
51 views

Heavy right skewed target

I scrapped Data science job postings of almost 1000 jobs for salary prediction. I wanted to bin the salary column and build a classification model from the job descriptions. But the (salary)target is ...
Sendhan's user avatar
  • 21
0 votes
0 answers
15 views

How to identiyfy issues in structure or missing values in netCDF-file that contains weather data?

during processing of weather data files (wind x,y 10m), i get the follwing error: height3' not present in all datasets and coords='different'. Either add 'height3' to datasets where it is missing or ...
otk's user avatar
  • 101
0 votes
0 answers
15 views

How to generate a netCDF weather forecast for a missing day based on yesterdays forecast?

as can be seen here met.no, the forcast is missing for 20210823. This forcast for each day is a 66 hour forecast, mening it will cover the next "missing" day. Not knowing much about netCDF ...
otk's user avatar
  • 101
0 votes
0 answers
16 views

Iterating more quickly, shorter feedback cycles, general workflow practices

(caveat: I've not written that much ML code yet) When I'm writing non-ML code I like to create short feedback loops to check the validity of my code. I use unit-tests or sometimes short helper tools (...
Niels Bom's user avatar
  • 101
1 vote
1 answer
63 views

Should I remove outliers before combining the dataset or after combining the dataset?

I am doing some exploratory data analysis on a custom dataset. I have 3 different data-frames: df1 belonging to class 0. df2 belonging to class 1. df3 belonging to class 2. I am doing k-means ...
Harshvardhan Uppaluru's user avatar
0 votes
1 answer
27 views

Academic name of dataset preparation method with hierarchical-learned labels? - E.g., cold→half-cooked→cooked

What's the name of the dataset preparation method indicating hierarchical ontologies? Assume photos of cold, half-cooked, and fully-cooked chickens. Annotate with temperature data. E.g., at current ...
Samuel Marks's user avatar
2 votes
2 answers
54 views

Handling Month-over-Month data in Regression Model

I have data similar to what you see in the picture. I want to use a RandomForest Regression model where I can use fields (excluding MONTH_END_DT and LOCATION_ID) to predict REVENUE_PER_UNIT. The idea/...
Larry Burholme's user avatar
0 votes
0 answers
30 views

How to work with tuples in pandas?

I have a dataset in a pandas dataframe where each cell contains a tuple (1,2), (3, 4)... I would like to write a function which adds the second number to a total_if_one column for each row.How does ...
Cory Blake's user avatar
0 votes
1 answer
18 views

'Unreported' value in data - what to use instead

I am compiling a list of all of our customers' revenue, which will be stored as an integer in our analytics app (ChartMogul). This revenue data is from Companies House in the UK. With small businesses ...
Jamie's user avatar
  • 1
0 votes
0 answers
18 views

how many different customer hierarchies exist in a customer data set

looking for some ideas that lead to a starting point on how to define all existing hierarchies in a data set. e.g. A customer database has many different customer hierarchies. how could one find the ...
Deathstar2008's user avatar
0 votes
0 answers
51 views

Mislabeled problem with hospital data

I am writing this post to ask or see if someone can help me with this problem in case you may have faced a similar situation. My problem has to do with a ML tool that I am trying to develop in a ...
Daniel's user avatar
  • 1
0 votes
0 answers
19 views

To Become a Data Analyst?

currently I am in year4 and also working as a product owner who would want to shift the occupation path to data analyst in banking, but the problem is I don't know where to start. May I have some ...
Hunter's user avatar
  • 1
0 votes
0 answers
23 views

Get accurate alias list of people without information on aliases

I'm working on a project where I have a large dataframe of paintings (from the Art500k dataset), each row corresponds with a painting, containing the author's name in the ...
me9hanics's user avatar
  • 113
0 votes
0 answers
12 views

Help with creating a panel dataset

I have some medical information of individuals spanning ten years which includes first name, last name, dob, and other variables (two/three that could help match observations i.e. state living, ...
GreenArrow's user avatar
1 vote
1 answer
112 views

Data Skewness is nan or inf

I have checked the Skewness of my data before applying a Log transformation using the next code : print("Skewness: %f" % df['Wind Speed (km/h)'].skew()) ...
baddy's user avatar
  • 165
0 votes
0 answers
23 views

Do outlier data in target variable cause a problem for GBM's?

I wonder if I should exclude outlier (legit data, not wrong readings) data from my dataset using gradient boosting. Let's say we try to predict water damage for regular houses and 99% of data is in 0-...
morqueatsz's user avatar
0 votes
0 answers
81 views

How to input or estimate missing text data?

My task requires 32 different columns with 25 beeing independent text data. Deleting nan values, or cutting columns which has less then 20% (of non NaNs) results in reducing dataset to less then 10% ...
Paweł B's user avatar
0 votes
0 answers
7 views

how to convert these string columns so that i can put the data to the regressor

https://drive.google.com/file/d/1yQR1xJvzWd2UGgGQS2bHhdywZEE_Nu0J/view?usp=sharing I have to change the name of the columns: city, batting_team, bowling_team, batsman, non_striker, bowler. These ...
gaurav najpande's user avatar
0 votes
1 answer
17 views

Dates countries joined/left international organisations

Does anyone know of a reliable data source containing the dates countries left or joined international organisations, for example,WTO, EU. etc? Ideally I'd like this going back to 1946. Even the year ...
James's user avatar
  • 121
0 votes
0 answers
25 views

How to parse the data from the json file describing via dataframe

This is the link to the json file I want to parse the data (resolve [a sentence] into its components and describe their syntactic roles.) Form this file into a dataframe describing: keys as columns ...
Charlie Max's user avatar
0 votes
2 answers
30 views

How to encode Income Type Ordinal Data into numbers?

I am doing a mini project on Credit card Approval Prediction. The Dataset I have taken is from Kaggle: https://www.kaggle.com/datasets/rikdifos/credit-card-approval-prediction Problem: I want to ...
Prajwal Dhage's user avatar
1 vote
1 answer
36 views

Removing specific phrases from textual data (R)

I have the following reddit posts and I would like to clean the posts and remove from the data the specific phrase "Click to expand", while keeping all other words within a post the same. <...
maldini1990's user avatar
0 votes
0 answers
56 views

Data Cleanup for Regression

I have a simple dataset of 1 output and 1 input and want to fit a linear regression to the dataset. The data has a certain level of noise to it (potentially driven by another input, which I will ...
felix_the_cat's user avatar
0 votes
0 answers
10 views

Segregation of a finance interview based on topic discussed

I have a video interview of 5 people, which i have transcribed to text (example corpus given below). Considering the fact that we know SPEAKER_00 is the interviewer and rest are guests. I want to know ...
Devansh Gupta's user avatar
0 votes
1 answer
21 views

What are some Models/Methods to reduce noise using environmental data?

I have a set of pressure datasets from a mechanical device that frequently moves around the country. I also have several sets of environmental data (Altitude, ambient temperature etc.) from those ...
PressureQuery's user avatar
0 votes
0 answers
15 views

Associating two variables in XGBoost model

I am trying to build a model that will predict a college football players probability of being drafted. I have multiple variables with different athletic measurements, but for the sake of example lets ...
user153943's user avatar
0 votes
0 answers
5 views

How to find which column makes a table unique?

I often work on community ecology data sets that I have to restructure to fit my template. They often have many columns: categorical columns describing the site such as ...
Alban's user avatar
  • 1
0 votes
0 answers
18 views

Handling Missing values in data

I have a Machine Learning task wherein I have to predict a target variable using an input. However, my dataset has a considerable percentage of missing values. What is the best approach to handle it? ...
Anwesh saha's user avatar
0 votes
0 answers
7 views

Machine log detection and removal

I don't know if there's a terminology for the following problem: Suppose I have a collection of data samples where each sample is a long text. A sample may contain machine logs. The DS problem is to ...
Tony's user avatar
  • 1
1 vote
1 answer
49 views

Natural Order should be maintained while ordinal encoding?

I am encoding my ordinal categorical values as VHigh=1, High=2,Med=3,Low=4. Am I doing correct? or order doesn't impact? If it impacts, how does it impact Decision Tree, Logistic Regression, SVM?
Pramod yadav's user avatar
0 votes
3 answers
35 views

Data preprocessing

I just want to know how to determine whether to remove the missing values or to impute them with mean , median or mode. I usually remove the missing values but it decreases the size of the dataset by ...
Yousaf Chaudhary's user avatar
0 votes
0 answers
12 views

PowerBI: Pivot questionnaire data that's returned record by record into one line per respondent

This is an oversimplification of the dataset I need to work with. Actually, I have to process over 60k records spanning over 27 questions. The data is downloaded through an API and every day new ...
Jeroen's user avatar
  • 1
0 votes
1 answer
130 views

How to handle multiple categories of ordinal data for clustering?

I wanted to ask about how multiple categories of ordinal data handled for the purposed of running some clustering. For instance, I have survey data where respondents answer the following questions: a. ...
Amatya's user avatar
  • 103

1
2 3 4 5
16