Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
3
votes
2answers
44 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
1answer
33 views

What are the best practises to decide whether a variable is categorical?

What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are ...
0
votes
1answer
36 views

What are the limits of capping ratios to [0-1]?

I work with some financial ratios. By construction those variables should be between 0 and 1. However, for some reasons I end up with values <0 and >1. These reasons include : reporting problems (...
0
votes
1answer
21 views

Should inputs be shuffled for training a model with expected time dependency?

Consider a prediction model with numerical inputs and outputs. Suppose data is inserted tick by tick, i.e., when new data is available it is inserted asynchronously. Current output depends on current ...
-3
votes
1answer
34 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
1
vote
1answer
15 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
2
votes
1answer
119 views

Filtering a panda dataframe in one line

I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process) Given the following ...
0
votes
0answers
10 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
6
votes
2answers
121 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
0
votes
3answers
780 views

Select samples from a dataframe in python [closed]

I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most. thanks
0
votes
1answer
226 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
0
votes
1answer
48 views

Handling unwanted negative numbers

With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a ...
2
votes
0answers
90 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
2
votes
1answer
60 views

About Hadley Wickham paper on tidy data

I read this paper and understood it very well, but I miss negative impact that will happen if we don’t fix the 5 types of messy data introduced in the paper (page 5), or even how fixing them will make ...
1
vote
3answers
183 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
1
vote
0answers
33 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
1
vote
1answer
35 views

Reduction of feature values

I have a data set of 700+ mil records with a feature that should yield good predictive power. The problem is that it has far more unique values than it should. The 10k+ unique values should map to ...
2
votes
2answers
80 views

File format where column names are repeated on each row

I have received a dataset in text file with the following format ...
0
votes
1answer
1k views

How to get mean of column using groupby() and another condition [closed]

For the following df = pd.DataFrame() df['1'] = 1,2,1,2,1,2 df['2'] = 3,6,5,4,7,8 df['3'] = 1,1,1,2,2,2 I want to do ...
0
votes
0answers
23 views

How do I export my data into .h5 format?

I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition. I need to prepare my own dataset in a format like this: ...
1
vote
1answer
100 views

Noise Elimination with majority vote filtering

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
0
votes
2answers
177 views
2
votes
1answer
1k views

Tips on how to preprocess data and outliers for churn analysis

I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions, 1) When I skewness and kurtosis for the normality test, two ...
2
votes
1answer
171 views

Data Extraction from images using NLP and ML [closed]

Hi i'm trying to extract data like name, planType, phone#, .. from images like insurance cards or licence cards with GoogleVision / Textract using some conditions but it does not extract the correct ...
0
votes
1answer
4k views

Creating new column in dataframe based on conditions in 2 other columns [closed]

I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns. As I have it written below, the column ...
2
votes
2answers
42 views

How do I find the count of a particular column, based on another column(date) using pandas?

I have a dataframe with 3 columns, such as SoldDate,Model and TotalSoldCount. How do I create a new column, 'CountSoldbyMonth' which will give the count of each of the many models sold monthly? <...
0
votes
1answer
28 views

CASE sentence for SQL query

I have a question abut MySQL I would like to handle a column with characters without changing the original DB. Below is the levels of the column named 'Maker'. 기아, 현대, BMW, VolksWagen, Porsche So ...
0
votes
1answer
32 views

How to transform many columns of TRUE/FALSE/NA to just TRUE/FALSE?

I have a dataframe that consists of a few columns of text, and then a bunch of columns that are TRUE/FALSE or NA (they were TRUE/FALSE but I left-joined them with ...
0
votes
2answers
199 views

Url string processing: what is the best way?

I have ~1000 different news websites and I scraped and saved all the internal url links for each website. For instance, the website dcgazette.com has a 2MB text file with associated urls: 1) https://...
1
vote
1answer
1k views

Feature engineering for categorical variables

I have some categorical variables in my dataset for a regression problem. 1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary ...
2
votes
1answer
38 views

How valuable is a categorical feature that has a predominant category over all other ones?

Is a categorical feature that has almost equally distributed in it's category more important or the one which one of it's category is predominant over all other ones? In data prepossessing step for "...
0
votes
1answer
18 views

how to export the data edited to a new excel file

The provided code imports the excel file and removes the not NaN values and store it in df2. Now, How do i save the changes to an another file. ...
0
votes
1answer
489 views

How to delete a row if a values in a column is not NaN

In the given code it displays only the rows without NaN values but i want only the rows with NaN values in a provided column everything has to be removed ...
2
votes
1answer
42 views

How to scale or standardize data that is mostly 0 (ranges from 0-1)?

I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each ...
2
votes
1answer
45 views

how to balance the data set with different number of observations

I am pretty new to ds field and recently I was working on a self project of the clustering model. My goal is to create clusters and see in each cluster what the common features are between customers. ...
4
votes
1answer
237 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
0
votes
1answer
580 views

Combine Pandas DataFrame Rows Based on Matching Data and Boolean

I have a Pandas DataFrame with sales data and columns for year, ISO week, price, ...
2
votes
1answer
36 views

Tool for clustering and cleansing data set

I have a large-ish data set (400K records) composed of two fields (both strings). I am looking for a tool that will enable me to cluster the data e.g. around the first column, either using exact ...
0
votes
1answer
18 views

Formatting multi-level variable width data for model

I would like to calculate a value based on the number of lines on outage at a given time. An outage consists of the two endpoints and the voltage of the line: ...
1
vote
1answer
87 views

If categorical variable has some hierarchy, should I just label them or split into dummy variables (One-Hot encode)?

I have a column which has 5 unique categories. There's a hierarchy between these categories (Best > good > OK/Not Sure > Bad > Worst) In this case, should I label them based on hierarchy like: ...
1
vote
1answer
82 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
73 views

Cleaning Excel Data in Merged Cells

I need to use an Excel file that have many merged cells with no specific order. I guess that the file creator thought that it would be more readable to merge the cells that have the same value. ...
1
vote
1answer
26 views

Data duplication optimization

I am working in python3 cleaning data. I have a large number of midi files scraped from a variety of sources using beautiful soup...
0
votes
2answers
139 views

Understanding missing values in dataset

I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) ...
1
vote
3answers
67 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
1
vote
0answers
14 views

Term for an identifier that has been superseded

Is there a 'proper' term for an ID (or IDs) that have been superseded by (or merged into) another ID? My use case: rsIDs are used by geneticists to refer to a SNP. These take the form of a string '...
0
votes
2answers
643 views

Tips for checking data integrity / data sanity?

I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets. I am interested in ...
0
votes
1answer
614 views

Script to convert non-numeric elements that are numbers to numeric

I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward. I downloaded some Google Play Store ...
3
votes
1answer
838 views

Jaccard Similarity with Binary Data

I have 5400 rows of data and 3211 columns of attributes. The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity ...
1
vote
2answers
22 views

Keeping part of a string in R [closed]

I have a dataframe with the following column city <- c("Sydney NSW", "Newcastle NSW", "Liverpool NSW", "Broken Hill NSW") I want to maintain everything prior ...

1 2 3
4
5
11