Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
41
votes
6answers
5k views

How can I transform names in a confidential data set to make it anonymous, but preserve some of the characteristics of the names?

Motivation I work with datasets that contain personally identifiable information (PII) and sometimes need to share part of a dataset with third parties, in a way that doesn't expose PII and subject ...
35
votes
7answers
4k views

Organized processes to clean data

From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis. Are there any best practices or processes for cleaning ...
27
votes
3answers
31k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
20
votes
3answers
13k views

is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of gather(), separate() and ...
18
votes
4answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
16
votes
1answer
110k views

removing strings after a certain character in a given text

I have a dataset like the one below. I want to remove all characters after the character ©. How can I do that in R? ...
14
votes
10answers
3k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform analysis of it. In this dataset, the gender field ...
14
votes
4answers
19k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
14
votes
5answers
11k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
14
votes
1answer
68k views

Convert a pandas column of int to timestamp datatype

I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately ...
12
votes
2answers
12k views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler. I know when Standard Scaler is applied. But in which scenario is Normalizer ...
11
votes
2answers
9k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
10
votes
2answers
81k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
9
votes
4answers
426 views

Math PhD (Nonlinear Programming) switching to Data Science?

I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
8
votes
2answers
59k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
8
votes
2answers
2k views

What are the best practices to anonymize user names in data?

I'm working on a project which asks fellow students to share their original text data for further analysis using data mining techniques, and, I think it would be appropriate to anonymize student names ...
8
votes
2answers
647 views

Fixing data inconsistencies

I'm trying to analyze some data I have but there is a lot of inconsistencies in my data. I have a SQL table that I'm trying to analyze. The table is a table of universities with the following ...
7
votes
1answer
9k views

Why should I normalize also the output data?

I'm new to data science and Neural Networks in general. Looking around many people say it is better to normalize the data between doing anything with the NN. I understand how normalizing the input ...
7
votes
2answers
10k views

One Hot Encoding vs Word Embeding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
7
votes
2answers
991 views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
7
votes
1answer
428 views

Neural Networks: How to prepare real world data to detect low probability events?

I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records ...
7
votes
1answer
12k views

How to extract paragraphs from text document?

I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like: Sample pdf file is present at http://www.heromotocorp.com/en-in/uploads/...
7
votes
1answer
199 views

Under what circumstance is lemmatization not an advisble step when working with text data?

Disregarding possible computational restraints, are there general applications where lemmatization would be a counterproductive step when analyzing text data? For example, would lemmatization be ...
6
votes
7answers
272 views

Good practices for manual modifications of data

More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed. When a fraction of data needs it, I write a script and ...
6
votes
2answers
197 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
6
votes
2answers
103 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
6
votes
2answers
47k views

How can I fill NaN values in a Pandas DataFrame in Python?

I am trying to learn data analysis and machine learning by trying out some problems. I found a competition "House prices" which is actually a playground competition. Since I am very new to this ...
6
votes
2answers
4k views

What methods can be used to detect anomalies in temporal texual data?

I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I ...
6
votes
1answer
2k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
5
votes
3answers
12k views

How can I perform stratified sampling for multi-label multi-class classification?

I am asking this question for few reasons: The dataset in hand is imbalanced I used below code ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
5
votes
1answer
3k views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
5
votes
1answer
296 views

Anonymizing Datasets

I would like to know what are the best practices for anonymizing datasets? Ideally I should be able to get the original data back after performing analysis on the anonymized dataset. Should I be ...
5
votes
1answer
3k views

Audio Analysis : Segment audio based on speaker recognition

I have audio clips of people being interviewed and am trying to split the audio clips using python such that all speech segments of the interviewee are outputted in one audio file (eg .wav format) &...
5
votes
1answer
166 views

Data scheduling for recommender

I do at the moment some data experiments with the Graphlab toolkit. I have at the first next SFrame, with the three columns: Users Items Rating The pair in the ...
5
votes
2answers
121 views

Amalgamating multiple datasets with different variables coding

I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables ...
4
votes
2answers
118 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
4
votes
3answers
169 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
4
votes
1answer
1k views

Creating training data

My task is to classify free text originated from customer complaints about our product. I have created a Taxonomy and have around 10 different categories. I've realized that these categories include ...
4
votes
1answer
83 views

How do I find the relevant features out of 11,000+ possibilities?

While working on Kaggle Competition, I ended up with 11,726 columns which are mostly "dummies" (one hot encoding). Is this too many? I know that we need to find out which features are relevant, but ...
4
votes
1answer
13k views

Pandas how to fill missing values in one column if the values in another column are equal

I have a dataframe where I need to fill in the missing values in one column (paid_date) by using the values from rows with the same value in a different column (id). There is guaranteed to be no more ...
4
votes
3answers
188 views

How to deal with count data in random forest

I am working on a classification model where my target class is a biased class with the class shape as 0 1 20694 101 Most of my features are the ...
4
votes
1answer
177 views

Can someone please explain what this sample function is upto?

So there is a function in Dino_Name_Generator at Deeplearning.ai notebook ...
4
votes
2answers
127 views

Why would you split your train data to compute a value on half of the data to then fill the Nan values on the other half?

I was checking a kernel written in python from the Bosch kaggle competition (kaggle link to python kernel) and I came across with a weird (at least to me) way to fill Nan values. The train data is ...
4
votes
2answers
182 views

Choosing data clustering method to visualize data

I'm working with a database about internally displaced persons in Colombia. All data are absolutes values, so I calculate the rate per 1000 people. I started to visualize all data using QGis. I ...
4
votes
1answer
38 views

Alignment of square nonorientable images/data

Another post where I don't know enough terminology to describe things efficiently. For the comments, please suggest some tags and keywords I can add to this post to make it better. Say I have a 2D ...
4
votes
0answers
44 views

How to filter this signal to get a heartbeat signal?

I am giving my first steps in data analysis, gathering/cleaning. To learn, I am trying to create a simple code that can detect heartbeats from color variations from the image coming from the camera ...
3
votes
3answers
2k views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
3
votes
2answers
3k views

Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing?

I am a professional java developer with 9 years of experience. Never worked with Python but, I do not have any restraints learning it. Still I need clarity on a couple of aspects. I hope if anyone ...

1
2 3 4 5
10