Questions tagged [data-cleaning]
Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.
761
questions
0
votes
0
answers
11
views
how many different customer hierarchies exist in a customer data set
looking for some ideas that lead to a starting point on how to define all existing hierarchies in a data set.
e.g. A customer database has many different customer hierarchies. how could one find the ...
0
votes
0
answers
33
views
Mislabeled problem with hospital data
I am writing this post to ask or see if someone can help me with this problem in case you may have faced a similar situation.
My problem has to do with a ML tool that I am trying to develop in a ...
- 1
0
votes
0
answers
18
views
To Become a Data Analyst?
currently I am in year4 and also working as a product owner who would want to shift the occupation path to data analyst in banking, but the problem is I don't know where to start. May I have some ...
- 1
0
votes
0
answers
13
views
Get accurate alias list of people without information on aliases
I'm working on a project where I have a large dataframe of paintings (from the Art500k dataset), each row corresponds with a painting, containing the author's name in the ...
- 13
0
votes
0
answers
11
views
Help with creating a panel dataset
I have some medical information of individuals spanning ten years which includes first name, last name, dob, and other variables (two/three that could help match observations i.e. state living, ...
1
vote
1
answer
27
views
Data Skewness is nan or inf
I have checked the Skewness of my data before applying a Log transformation using the next code :
print("Skewness: %f" % df['Wind Speed (km/h)'].skew())
...
- 165
0
votes
0
answers
13
views
Do outlier data in target variable cause a problem for GBM's?
I wonder if I should exclude outlier (legit data, not wrong readings) data from my dataset using gradient boosting.
Let's say we try to predict water damage for regular houses and 99% of data is in 0-...
- 25
0
votes
0
answers
76
views
How to input or estimate missing text data?
My task requires 32 different columns with 25 beeing independent text data.
Deleting nan values, or cutting columns which has less then 20% (of non NaNs) results in reducing dataset to less then 10% ...
- 1
0
votes
0
answers
7
views
how to convert these string columns so that i can put the data to the regressor
https://drive.google.com/file/d/1yQR1xJvzWd2UGgGQS2bHhdywZEE_Nu0J/view?usp=sharing
I have to change the name of the columns: city, batting_team, bowling_team, batsman, non_striker, bowler. These ...
0
votes
1
answer
17
views
Dates countries joined/left international organisations
Does anyone know of a reliable data source containing the dates countries left or joined international organisations, for example,WTO, EU. etc? Ideally I'd like this going back to 1946. Even the year ...
- 121
0
votes
0
answers
19
views
How to parse the data from the json file describing via dataframe
This is the link to the json file I want to parse the data (resolve [a sentence] into its components and describe their syntactic roles.)
Form this file into a dataframe describing:
keys as columns
...
0
votes
2
answers
19
views
How to encode Income Type Ordinal Data into numbers?
I am doing a mini project on Credit card Approval Prediction. The Dataset I have taken is from Kaggle: https://www.kaggle.com/datasets/rikdifos/credit-card-approval-prediction
Problem: I want to ...
1
vote
1
answer
31
views
Removing specific phrases from textual data (R)
I have the following reddit posts and I would like to clean the posts and remove from the data the specific phrase "Click to expand", while keeping all other words within a post the same.
<...
- 127
0
votes
0
answers
41
views
Data Cleanup for Regression
I have a simple dataset of 1 output and 1 input and want to fit a linear regression to the dataset.
The data has a certain level of noise to it (potentially driven by another input, which I will ...
0
votes
0
answers
8
views
Segregation of a finance interview based on topic discussed
I have a video interview of 5 people, which i have transcribed to text (example corpus given below). Considering the fact that we know SPEAKER_00 is the interviewer and rest are guests. I want to know ...
0
votes
1
answer
18
views
What are some Models/Methods to reduce noise using environmental data?
I have a set of pressure datasets from a mechanical device that frequently moves around the country. I also have several sets of environmental data (Altitude, ambient temperature etc.) from those ...
0
votes
0
answers
14
views
Associating two variables in XGBoost model
I am trying to build a model that will predict a college football players probability of being drafted. I have multiple variables with different athletic measurements, but for the sake of example lets ...
0
votes
0
answers
4
views
How to find which column makes a table unique?
I often work on community ecology data sets that I have to restructure to fit my template. They often have many columns:
categorical columns describing the site such as ...
- 1
0
votes
0
answers
18
views
Handling Missing values in data
I have a Machine Learning task wherein I have to predict a target variable using an input. However, my dataset has a considerable percentage of missing values. What is the best approach to handle it?
...
0
votes
0
answers
5
views
Machine log detection and removal
I don't know if there's a terminology for the following problem:
Suppose I have a collection of data samples where each sample is a long text. A sample may contain machine logs. The DS problem is to ...
- 1
0
votes
1
answer
33
views
Natural Order should be maintained while ordinal encoding?
I am encoding my ordinal categorical values as VHigh=1, High=2,Med=3,Low=4. Am I doing correct? or order doesn't impact?
If it impacts, how does it impact Decision Tree, Logistic Regression, SVM?
- 11
0
votes
3
answers
32
views
Data preprocessing
I just want to know how to determine whether to remove the missing values or to impute them with mean , median or mode. I usually remove the missing values but it decreases the size of the dataset by ...
0
votes
0
answers
12
views
PowerBI: Pivot questionnaire data that's returned record by record into one line per respondent
This is an oversimplification of the dataset I need to work with. Actually, I have to process over 60k records spanning over 27 questions. The data is downloaded through an API and every day new ...
- 1
0
votes
1
answer
55
views
How to handle multiple categories of ordinal data for clustering?
I wanted to ask about how multiple categories of ordinal data handled for the purposed of running some clustering.
For instance, I have survey data where respondents answer the following questions:
a. ...
- 103
0
votes
0
answers
8
views
I want to import my states legislation into a database. I need advice in parsing large HTML, TXT into an importable form. It is Large PDF initialy
My data consists of large pdf files containing state legislation. I can convert the pdf to xml, txt or html, and I can manually go through, identify the tables and write a script to extract only the ...
- 1
0
votes
1
answer
46
views
Finding outlier for millions of data that can be used cluster-based algorithm
I have millions of data that each have many features. But as long as the value of a feature is not in the acceptable range, that data will be considered an outlier. And I need to find the acceptable ...
- 3
0
votes
1
answer
45
views
Feature Selection - determining the significance of imbalanced categorical data column
I have a dataset with a categorical column that contains three categories. One of the categories represents 98% of the data, while the remaining 2% are distributed between the other two categories, ...
0
votes
0
answers
33
views
How can I improve accuracy of my ensemble (or anywhere in the code where I can increase accuracy)?
I am pretty new to machine learning, so if my code is not good, please bear with me.
...
0
votes
1
answer
25
views
Clustering and grouping noisy time series fuel data
I have a time series of fuel data which are split into distinct clusters (as shown in the image below. I'm looking at the green line). The clusters almost always have a descending gradient. I would ...
0
votes
0
answers
24
views
How to predict continous values using resnet regression model
I'm making a resnet regression model to predict force plate data using smartinsole data. so my data input is smartinsole data and the force plate is the label. below is an example of my data sample.
<...
- 31
0
votes
1
answer
343
views
Parsing Python List values from column in pandas data frame??? Data cleaning help me please
I am new to the community and data science. I have found this hub to be of extreme value and hope to be an additive contributor one day.
VS Code: Python: Pandas Question
I am working with a data set ...
0
votes
1
answer
29
views
How should I proceed with this type of dataset and requirement?
Suppose I have a dataset where I want to give input of up-to 30 lines (can be variable with min 5 available) these lines are timestamps of every minute and want my model to predict output of next 5 ...
0
votes
1
answer
65
views
How to handle ID variables when splitting data for machine learning?
I'm new to machine learning, and I'm working with some international head-to-head sports competition data. I used relational data creation techniques in tidyverse to join several data sources to ...
- 1
2
votes
2
answers
87
views
Why does pandas.dataframe.info() not update when I delete the first row of a dataset?
I am working with a dataset that comes in with nonsense field names in the first row, with the actual field names in the second row.
Currently I'm using this script:
...
- 165
0
votes
1
answer
1k
views
Pandas: To find duplicate columns
I have a dataframe with different dtypes like int, float, object, datatime etc. I am performing data cleaning, to list or find ...
- 115
0
votes
0
answers
6
views
How to construct Wiener Filter from powerspectrum?
I am trying to construct a Wiener Filter, to filter the ratio of the peak from the cross-correlation function, between a galaxy spectra and a template spectra, with the peak of the auto-correlation ...
0
votes
0
answers
13
views
Cloud computing solutions for processing big datasets/ML development?
I apologize in advance if my question is vague - I will edit it/follow up if it requires more context. I would also like to add that my experience is with small-ish datasets that can be run on lower-...
0
votes
0
answers
15
views
How to analyze 2 NLP datasets and understand their differences
I have 2 test sets (let's call them test set #1 and test set #2), and 2 Language models (let's call them model A and model B).
The issue I'm facing is that model A is performing better than model B on ...
- 1,004
0
votes
1
answer
173
views
What's the best approach to dealing with missing data in a dataset?
I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
- 1
0
votes
0
answers
27
views
Read from database every time a new record goes through a Spark pipeline
I'm using Spark to filter and transform every new record that goes through a data stream, the problem is that for each new record I need to read a table from a Cassandra database in order to have the ...
0
votes
0
answers
7
views
What should be the default value for missing ordinal variable
I want to rerank items based on shipping timelines. But I get shipping info from upstream service only when it is less than 4 days. We don't show user when the shipping timeline is more than 4 days. I ...
- 101
0
votes
2
answers
30
views
Data imputation for heavily missing features
I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
0
votes
0
answers
16
views
Having trouble manipulating data to make better dashboard
I have a test dataset which has has 39 columns. Index is given below:
Sample data is as follows:
Some columns are omitted due to privacy concerns. The colored cells in the index belong to one type. ...
0
votes
3
answers
79
views
0
votes
0
answers
19
views
Time series : add constants for each time series
I have N time series of shape (30*36) (time step, feature). For each time series 500 parameters that can be seen as the history of the time series have to be added. ...
- 29
0
votes
1
answer
23
views
Best practice for variables that only have answer if yes in previous column
I currently have a dataset that consists of survey data that has several columns that have answers dependent on the previous question. For example, I may have a question that says "Did you take ...
- 23
0
votes
0
answers
53
views
How can I convert YYYYMM-WN format to YYYYWW format?
I am working on cleaning a time series data that has a weekly frequency. However, the format of the date is a bit convoluted. Here's how it looks like
...
- 63
0
votes
0
answers
12
views
How to use text-to-columns or an equivalent operation on records in 1 field with varying lengths & no clear delimiters built into them
If this question in inappropriate for Data Science Stack Exchange, but there is another one within this stack it would be appropriate for, please let me know! It looks borderline to me.
A member of my ...
- 157
0
votes
2
answers
172
views
Data Preparation for next word prediction
In most places, I have seen that when preparing the training data and label for next-word prediction from the corpus one uses a fixed window size say of length 4, and then scans the subsequences of ...
- 101
0
votes
0
answers
14
views
How to store (lab) data in everyday cases
I'm (let's say half) a data scientist within an engineering group and as I'm the only one familiar with storing, manipulating, using and plotting data (other than only ...
- 540