Skip to main content

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
44 votes
6 answers
7k views

How can I transform names in a confidential data set to make it anonymous, but preserve some of the characteristics of the names?

Motivation I work with datasets that contain personally identifiable information (PII) and sometimes need to share part of a dataset with third parties, in a way that doesn't expose PII and subject ...
Air's user avatar
  • 822
37 votes
7 answers
5k views

Organized processes to clean data

From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis. Are there any best practices or processes for cleaning ...
Jay Godse's user avatar
  • 471
31 votes
3 answers
33k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
William Falcon's user avatar
28 votes
2 answers
175k views

Removing strings after a certain character in a given text

I have a dataset like the one below. I would like to remove all characters after the character ©. How can I do that in R? ...
Hamideh's user avatar
  • 940
25 votes
4 answers
16k views

Is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
cpumar's user avatar
  • 807
23 votes
5 answers
4k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
Amir Ali Akbari's user avatar
22 votes
2 answers
92k views

Convert a pandas column of int to timestamp datatype

I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately ...
Austin Capobianco's user avatar
22 votes
3 answers
49k views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler. I know when Standard Scaler is applied. But in which scenario is Normalizer ...
Heisenbug's user avatar
  • 411
20 votes
5 answers
17k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
AffableAmbler's user avatar
16 votes
4 answers
27k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
Stéphanie C's user avatar
16 votes
3 answers
15k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
DavideChicco.it's user avatar
15 votes
3 answers
25k views

How can I perform stratified sampling for multi-label multi-class classification?

I am asking this question for few reasons: The dataset in hand is imbalanced I used below code ...
Divyanshu Shekhar's user avatar
15 votes
1 answer
23k views

Why should I normalize also the output data?

I'm new to data science and Neural Networks in general. Looking around, many people say it is better to normalize the data before doing anything with the NN. I understand how normalizing the input ...
Euler_Salter's user avatar
14 votes
10 answers
5k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it. In this dataset, the gender ...
nick012000's user avatar
13 votes
1 answer
4k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
jmvllt's user avatar
  • 619
12 votes
5 answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
rocksNwaves's user avatar
12 votes
2 answers
92k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
Sanoj's user avatar
  • 251
10 votes
4 answers
2k views

Math PhD (Nonlinear Programming) switching to Data Science?

I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
John D's user avatar
  • 123
10 votes
3 answers
8k views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
Greg Rosen's user avatar
9 votes
1 answer
17k views

How to extract paragraphs from text document?

I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like: Sample pdf file is here FORWARD-LOOKING STATEMENTS In this Annual ...
Sanjeev's user avatar
  • 191
8 votes
2 answers
70k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
uharsha33's user avatar
  • 331
8 votes
1 answer
13k views

Encoding with OrdinalEncoder : how to give levels as user input?

I am trying to do ordinal encoding using: from sklearn.preprocessing import OrdinalEncoder I will try to explain my problem with a simple dataset. ...
Ayush Ranjan's user avatar
8 votes
1 answer
30k views

how to check all values in particular column has same data type or not?

I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. ...
Kiran's user avatar
  • 195
8 votes
2 answers
3k views

What are the best practices to anonymize user names in data?

I'm working on a project which asks fellow students to share their original text data for further analysis using data mining techniques, and, I think it would be appropriate to anonymize student names ...
xtian's user avatar
  • 193
8 votes
2 answers
3k views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
Muhammad Ali's user avatar
  • 2,487
8 votes
2 answers
695 views

Fixing data inconsistencies

I'm trying to analyze some data I have but there is a lot of inconsistencies in my data. I have a SQL table that I'm trying to analyze. The table is a table of universities with the following ...
bl0b's user avatar
  • 183
8 votes
2 answers
8k views

Audio Analysis : Segment audio based on speaker recognition

I have audio clips of people being interviewed and am trying to split the audio clips using python such that all speech segments of the interviewee are outputted in one audio file (eg .wav format) &...
aamir23's user avatar
  • 181
7 votes
4 answers
3k views

Does label encoding an entire dataset cause data leakage?

I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the ...
kaylani2's user avatar
7 votes
1 answer
21k views

Pandas how to fill missing values in one column if the values in another column are equal

I have a dataframe where I need to fill in the missing values in one column (paid_date) by using the values from rows with the same value in a different column (id). There is guaranteed to be no more ...
rob's user avatar
  • 73
7 votes
1 answer
500 views

Neural Networks: How to prepare real world data to detect low probability events?

I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records ...
Ans Stru's user avatar
7 votes
1 answer
1k views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this data frame (df): col1 col2 col3 col4 A f 5 g D er 2e sd F ...
randomizer0000's user avatar
7 votes
1 answer
859 views

Under what circumstance is lemmatization not an advisble step when working with text data?

Disregarding possible computational restraints, are there general applications where lemmatization would be a counterproductive step when analyzing text data? For example, would lemmatization be ...
Zer0k's user avatar
  • 155
6 votes
7 answers
500 views

Good practices for manual modifications of data

More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed. When a fraction of data needs it, I write a script and ...
Piotr Migdal's user avatar
6 votes
2 answers
225 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
Jeffrey04's user avatar
  • 241
6 votes
3 answers
76k views

How can I fill NaN values in a Pandas DataFrame in Python?

I am trying to learn data analysis and machine learning by trying out some problems. I found a competition "House prices" which is actually a playground competition. Since I am very new to this ...
Ahmed Dhanani's user avatar
6 votes
2 answers
899 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
Eileen's user avatar
  • 61
6 votes
1 answer
89k views

TypeError: float() argument must be a string or a number, not 'function'

I am trying to clean the data. But I don't know how to remove a function from a column in data frame. At row number 473 it show column N has a function . How it should be filtered out ?
Abhishek Priyankar's user avatar
6 votes
1 answer
13k views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
stats_nerd's user avatar
6 votes
2 answers
4k views

What methods can be used to detect anomalies in temporal texual data?

I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I ...
Sanjeev Rathor's user avatar
5 votes
3 answers
4k views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
ZelelB's user avatar
  • 1,057
5 votes
4 answers
5k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
Abhishek Sharma's user avatar
5 votes
1 answer
420 views

Anonymizing Datasets

I would like to know what are the best practices for anonymizing datasets? Ideally I should be able to get the original data back after performing analysis on the anonymized dataset. Should I be ...
Krishnaa's user avatar
  • 151
5 votes
1 answer
101 views

How do I find the relevant features out of 11,000+ possibilities?

While working on Kaggle Competition, I ended up with 11,726 columns which are mostly "dummies" (one hot encoding). Is this too many? I know that we need to find out which features are relevant, but ...
B Seven's user avatar
  • 292
5 votes
2 answers
6k views

When to remove outlier in preparing features for machine learning algorithm

I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, ...
KevinKim's user avatar
  • 635
5 votes
2 answers
16k views

Merging dataframes in Pandas is taking a surprisingly long time

I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
m13op22's user avatar
  • 382
5 votes
1 answer
210 views

Data scheduling for recommender

I do at the moment some data experiments with the Graphlab toolkit. I have at the first next SFrame, with the three columns: Users Items Rating The pair in the ...
Guforu's user avatar
  • 313
5 votes
2 answers
141 views

Amalgamating multiple datasets with different variables coding

I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables ...
user92953's user avatar
4 votes
4 answers
67k views

Using pandas, check a column for matching text and update new column if TRUE

My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE. From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, ...
RustyNails's user avatar
4 votes
2 answers
245 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
Ernesto Lopez Fune's user avatar
4 votes
3 answers
12k views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
CJL's user avatar
  • 143

1
2 3 4 5
16