Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
43
votes
6answers
5k views

How can I transform names in a confidential data set to make it anonymous, but preserve some of the characteristics of the names?

Motivation I work with datasets that contain personally identifiable information (PII) and sometimes need to share part of a dataset with third parties, in a way that doesn't expose PII and subject ...
35
votes
7answers
5k views

Organized processes to clean data

From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis. Are there any best practices or processes for cleaning ...
29
votes
3answers
32k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
22
votes
4answers
15k views

Is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
21
votes
5answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
20
votes
2answers
147k views

Removing strings after a certain character in a given text

I have a dataset like the one below. I would like to remove all characters after the character ©. How can I do that in R? ...
17
votes
5answers
15k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
17
votes
1answer
83k views

Convert a pandas column of int to timestamp datatype

I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately ...
16
votes
4answers
25k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
16
votes
3answers
28k views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler. I know when Standard Scaler is applied. But in which scenario is Normalizer ...
16
votes
2answers
12k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
14
votes
10answers
4k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it. In this dataset, the gender ...
11
votes
3answers
20k views

How can I perform stratified sampling for multi-label multi-class classification?

I am asking this question for few reasons: The dataset in hand is imbalanced I used below code ...
11
votes
1answer
16k views

Why should I normalize also the output data?

I'm new to data science and Neural Networks in general. Looking around many people say it is better to normalize the data between doing anything with the NN. I understand how normalizing the input ...
11
votes
2answers
88k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
11
votes
1answer
2k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
9
votes
4answers
926 views

Math PhD (Nonlinear Programming) switching to Data Science?

I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
8
votes
2answers
69k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
8
votes
3answers
5k views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
8
votes
2answers
2k views

What are the best practices to anonymize user names in data?

I'm working on a project which asks fellow students to share their original text data for further analysis using data mining techniques, and, I think it would be appropriate to anonymize student names ...
8
votes
2answers
2k views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
8
votes
2answers
675 views

Fixing data inconsistencies

I'm trying to analyze some data I have but there is a lot of inconsistencies in my data. I have a SQL table that I'm trying to analyze. The table is a table of universities with the following ...
8
votes
1answer
15k views

How to extract paragraphs from text document?

I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like: Sample pdf file is here FORWARD-LOOKING STATEMENTS In this Annual ...
7
votes
1answer
451 views

Neural Networks: How to prepare real world data to detect low probability events?

I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records ...
7
votes
2answers
5k views

Audio Analysis : Segment audio based on speaker recognition

I have audio clips of people being interviewed and am trying to split the audio clips using python such that all speech segments of the interviewee are outputted in one audio file (eg .wav format) &...
7
votes
1answer
403 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this data frame (df): col1 col2 col3 col4 A f 5 g D er 2e sd F ...
7
votes
1answer
444 views

Under what circumstance is lemmatization not an advisble step when working with text data?

Disregarding possible computational restraints, are there general applications where lemmatization would be a counterproductive step when analyzing text data? For example, would lemmatization be ...
6
votes
7answers
306 views

Good practices for manual modifications of data

More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed. When a fraction of data needs it, I write a script and ...
6
votes
3answers
2k views

Does label encoding an entire dataset cause data leakage?

I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the ...
6
votes
2answers
206 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
6
votes
3answers
72k views

How can I fill NaN values in a Pandas DataFrame in Python?

I am trying to learn data analysis and machine learning by trying out some problems. I found a competition "House prices" which is actually a playground competition. Since I am very new to this ...
6
votes
2answers
556 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
6
votes
1answer
75k views

TypeError: float() argument must be a string or a number, not 'function'

I am trying to clean the data. But I don't know how to remove a function from a column in data frame. At row number 473 it show column N has a function . How it should be filtered out ?
6
votes
2answers
4k views

What methods can be used to detect anomalies in temporal texual data?

I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I ...
5
votes
4answers
4k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
5
votes
1answer
3k views

Encoding with OrdinalEncoder : how to give levels as user input?

I am trying to do ordinal encoding using: from sklearn.preprocessing import OrdinalEncoder I will try to explain my problem with a simple dataset. ...
5
votes
1answer
13k views

how to check all values in particular column has same data type or not?

I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. ...
5
votes
1answer
362 views

Anonymizing Datasets

I would like to know what are the best practices for anonymizing datasets? Ideally I should be able to get the original data back after performing analysis on the anonymized dataset. Should I be ...
5
votes
1answer
89 views

How do I find the relevant features out of 11,000+ possibilities?

While working on Kaggle Competition, I ended up with 11,726 columns which are mostly "dummies" (one hot encoding). Is this too many? I know that we need to find out which features are relevant, but ...
5
votes
1answer
18k views

Pandas how to fill missing values in one column if the values in another column are equal

I have a dataframe where I need to fill in the missing values in one column (paid_date) by using the values from rows with the same value in a different column (id). There is guaranteed to be no more ...
5
votes
1answer
176 views

Data scheduling for recommender

I do at the moment some data experiments with the Graphlab toolkit. I have at the first next SFrame, with the three columns: Users Items Rating The pair in the ...
5
votes
2answers
129 views

Amalgamating multiple datasets with different variables coding

I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables ...
4
votes
3answers
3k views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
4
votes
4answers
63k views

Using pandas, check a column for matching text and update new column if TRUE

My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE. From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, ...
4
votes
2answers
170 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
4
votes
3answers
182 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
4
votes
1answer
24k views

How to count grouped occurrences?

My dataset has 2 columns: 1) Gender( that has 1 or 2) 2) Rating1(That has 3 ratings (1,2,3)) rating1: 1,2,3,3,2 gender: 2,1,2,2,1 I want output as: I have ...
4
votes
2answers
7k views

Pandas Conditional Fill NaN Forward/Backward

Updated 22 Oct. 2018: I have the following dataset: ...
4
votes
2answers
443 views

Tidy data in panda dataframe

There is a quite famous article by H. Wickham, Tidy data, where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he ...

1
2 3 4 5
13