Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
37 votes
7 answers
5k views

Organized processes to clean data

From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis. Are there any best practices or processes for cleaning ...
Jay Godse's user avatar
  • 471
20 votes
5 answers
17k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
AffableAmbler's user avatar
4 votes
1 answer
1k views

Creating training data

My task is to classify free text originated from customer complaints about our product. I have created a Taxonomy and have around 10 different categories. I've realized that these categories include ...
gogasca's user avatar
  • 749
1 vote
1 answer
137 views

Grouping profiles strings having the same words, but occurring out of order

I have a dataframe containing a column of profile types, which looks like this: ...
Erin's user avatar
  • 81
44 votes
6 answers
7k views

How can I transform names in a confidential data set to make it anonymous, but preserve some of the characteristics of the names?

Motivation I work with datasets that contain personally identifiable information (PII) and sometimes need to share part of a dataset with third parties, in a way that doesn't expose PII and subject ...
Air's user avatar
  • 822
31 votes
3 answers
33k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
William Falcon's user avatar
28 votes
2 answers
175k views

Removing strings after a certain character in a given text

I have a dataset like the one below. I would like to remove all characters after the character ©. How can I do that in R? ...
Hamideh's user avatar
  • 940
23 votes
5 answers
4k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
Amir Ali Akbari's user avatar
16 votes
4 answers
27k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
Stéphanie C's user avatar
16 votes
2 answers
15k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
DavideChicco.it's user avatar
15 votes
1 answer
23k views

Why should I normalize also the output data?

I'm new to data science and Neural Networks in general. Looking around, many people say it is better to normalize the data before doing anything with the NN. I understand how normalizing the input ...
Euler_Salter's user avatar
13 votes
1 answer
4k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
jmvllt's user avatar
  • 619
12 votes
5 answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
rocksNwaves's user avatar
10 votes
4 answers
2k views

Math PhD (Nonlinear Programming) switching to Data Science?

I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
John D's user avatar
  • 123
5 votes
1 answer
210 views

Data scheduling for recommender

I do at the moment some data experiments with the Graphlab toolkit. I have at the first next SFrame, with the three columns: Users Items Rating The pair in the ...
Guforu's user avatar
  • 313
5 votes
4 answers
5k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
Abhishek Sharma's user avatar
5 votes
2 answers
6k views

When to remove outlier in preparing features for machine learning algorithm

I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, ...
KevinKim's user avatar
  • 635
4 votes
1 answer
48 views

Alignment of square nonorientable images/data

Another post where I don't know enough terminology to describe things efficiently. For the comments, please suggest some tags and keywords I can add to this post to make it better. Say I have a 2D ...
Mark's user avatar
  • 213
3 votes
1 answer
2k views

Tips on how to preprocess data and outliers for churn analysis

I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions, 1) When I skewness and kurtosis for the normality test, two ...
Jason's user avatar
  • 49
3 votes
2 answers
2k views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
José Augusto's user avatar
3 votes
1 answer
644 views

Find most important features which differentiate two sets

Assume that we have two small sets of feature vectors (each feature vector representing an item). What is a good way of finding which features have the biggest difference (in distribution) between the ...
cinnamonAsh's user avatar
2 votes
1 answer
429 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
Edward Garemo's user avatar
2 votes
1 answer
451 views

Dealing with categorical variables in regression problems which method to use?

Usually if I have regression problem and my initial dataset contains categorical variables like : column 1: Math Science Science English I would convert this ...
baddy's user avatar
  • 165
2 votes
4 answers
7k views

Handling a feature containing multiple values

I have a dataset in following format: Movie ID | Actors | Director | language | ReleaseYear | Genre 1 | Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran | Vidhu Vinod ...
romitheguru's user avatar
1 vote
0 answers
435 views

How to combine data from multiple Google Trends queries effectively?

As you might know, Google Trends works by normalising a random sample of the search term data, with the sample changing at least once per day, from my experience. This is not an issue for western ...
PhilipTsv's user avatar
1 vote
3 answers
696 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
Rohit Gavval's user avatar
0 votes
1 answer
3k views

recognizing the correct word & "Set type is unordered"-error in python-pandas

My Data Set (CSV): ...
Jsmoka's user avatar
  • 13
0 votes
1 answer
9 views

Reducing emails token count preprocessing for Large Email Datasets - Feeding LLMs

I have a large email dataset in .txt format and want to feed LLMs (like Gemini and ChatGPT) to provide answers based on email content. The token count for my email data is very high (~1M for 1K emails)...
Rafael Borja's user avatar
-1 votes
1 answer
489 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
John Constantine's user avatar