Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
0answers
7 views

Recognizing emerging topic within ongoing topic

I have been collecting a large amount of tweets from Twitter for a few weeks related to a series of specific keywords, and would like to address a specific problem. Say I collected all tweets ...
0
votes
3answers
29 views

What is the better to pre-process data, MS-Excel or Python/R [on hold]

While handling data, many anomalies occur that might change the results. For example, outliers, missing values, etc. So in this context, what is the better tool to pre process data? A visual tool like ...
0
votes
2answers
20 views

Pandas: Find number of instances in another column [on hold]

I have a data frame like this: PERSON PARENT a null b a c a d c e null I know the names of everyone and who is that person's parent is. ...
0
votes
0answers
10 views

How to get a sample/summarised data from a table for data analysis?

I am currently working on EHR data analysis. I have a measurement table which has information lab test information of the patients. However the record count is huge. These are table count in my EHR ...
0
votes
0answers
12 views

Optimal practices to group data by Customer ID for churn prediction

Here's a quite common problem and I read a couple of questions/answers on it, however I still having my doubts about what are the best practices for grouping data by Customer ID for churn prediction. ...
0
votes
0answers
11 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
1answer
37 views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
17 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
0answers
17 views

How to deal with similar feature values but each indicates to a different information?

If I have a feature with replicated values but each of these values indicates a different piece of information. example: feature 'street name' with value 'A' which some of these 'A's are for Boston ...
0
votes
0answers
4 views

Pandas dataframe manipulation [migrated]

I am working on cleaning a huge dataset using python. I have a data frame with 20,000 rows that looks like this: ...
0
votes
0answers
16 views

Data Transformation Tips for xgboost's XGBClassifier

I have this X_train and test distribution for the 4 features 'X', 'Y', 'TX' and 'TY'. I realize the range of the distribution is widely varying .. Can you suggest a good way to clean/ transform that ...
8
votes
4answers
198 views

Math PhD (Nonlinear Programming) switching to Data Science?

I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is ...
0
votes
0answers
12 views

How do I clean the data for cluster analysis using sas miner

Hi I am looking to run some associative and clustering analyses using this dataset on kaggle. All the methods on Kaggle is in R and Python. Any ideas on how to clean and deal with the missing values? ...
1
vote
1answer
31 views

How to handle out of scope value?

I have a data set in which the score column has to be between 0 to 100 and the subject column has to be one of ['Math','Science','English']. However, my data set ...
1
vote
1answer
52 views

how to check all values in particular column has same data type or not?

I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. ...
0
votes
0answers
15 views

Averaging over Irregular Intervals

Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation? Specifically, suppose we are ...
0
votes
0answers
8 views

Where can I get an untokenized version of GLUE's SST-2 dataset?

On the GLUE faq, they say: Similarly, for SST, the data provided is already tokenized. We're working on obtaining a version that is not tokenized. Feel free to train on other distributions of ...
2
votes
0answers
37 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
0
votes
0answers
21 views

How to handle a data set with large number (about 75%) of binary variables?

I am doing a research right now and want to classify (predict) churns of costumers using machine learning. My data set consists of about 500,000 observations with 20 variables: 15 are binary, 2 ...
1
vote
1answer
21 views

Remove rows that are too much alike not to be duplicates

I have a dataset of real estate advertisements. Several of the lines are about the same real estate property so it's full of duplicates that aren't exactly the same. What would be the best methods to ...
1
vote
3answers
23 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
0answers
7 views

Transforming target from object array to integer array to use sparse_categorical_crossentropy for class prediction

I want to do a neural network to predict to which loan class does a borower pertains. There are 6 classes [ A, B, C, D, E, F]. I tried to get rid of the NAs and ...
0
votes
1answer
22 views

Grouping similar rows to detect duplicates

I have a dataset containing all the real estate ads for sale in the process of publishing a city: ...
1
vote
2answers
47 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
1
vote
1answer
30 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
0
votes
0answers
14 views

How to convert a non gaussian distribution into a gaussian destribution?

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my ...
0
votes
0answers
19 views

Model Checking in Multiple Imputation

Trying to understand how to incorporate Multivariate Imputation by Chained Equation (MICE) for handling missing data in python. I know there are a few libraries that can implement MICE on a dataset ...
1
vote
0answers
23 views

Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, ...
3
votes
2answers
43 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
1answer
26 views

What are the best practises to decide whether a variable is categorical?

What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are ...
0
votes
1answer
22 views

What are the limits of capping ratios to [0-1]?

I work with some financial ratios. By construction those variables should be between 0 and 1. However, for some reasons I end up with values <0 and >1. These reasons include : reporting problems (...
0
votes
1answer
17 views

Should inputs be shuffled for training a model with expected time dependency?

Consider a prediction model with numerical inputs and outputs. Suppose data is inserted tick by tick, i.e., when new data is available it is inserted asynchronously. Current output depends on current ...
0
votes
1answer
26 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
1
vote
1answer
11 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
2
votes
1answer
50 views

Filtering a panda dataframe in one line

I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process) Given the following ...
0
votes
0answers
8 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
6
votes
2answers
32 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
0
votes
3answers
50 views

Select samples from a dataframe in python [closed]

I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most. thanks
0
votes
0answers
22 views

Real-time Time-Series Error Correction

I have some sensors, each of which generates data points at mostly-regular intervals. So for each sensor I have a time series. I have to correct measurement errors from the sensors, replacing them ...
0
votes
1answer
60 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
0
votes
1answer
25 views

Handling unwanted negative numbers

With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a ...
2
votes
0answers
37 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
1
vote
1answer
30 views

About Hadley Wickham paper on tidy data

I read this paper and understood it very well, but I miss negative impact that will happen if we don’t fix the 5 types of messy data introduced in the paper (page 5), or even how fixing them will make ...
0
votes
3answers
48 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
0
votes
0answers
21 views

Intent Classification with imbalanced dataset

I am trying to train my ML model to classify 13 intents. Prior to training the model, the training data was sufficiently accurate. Here are the imbalanced training data sizes per class (class_num: ...
1
vote
0answers
15 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
0
votes
1answer
24 views

Reduction of feature values

I have a data set of 700+ mil records with a feature that should yield good predictive power. The problem is that it has far more unique values than it should. The 10k+ unique values should map to ...
0
votes
0answers
19 views

Handling Multiple Classes in Categorical variables and Modeling help

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the ...
0
votes
0answers
19 views

Extract user input from a filled document form

I have some images of filled in documents with the same form scanned with different angle rotation, different luminosity, different noise etc, and also the image of empty form scanned correctly. My ...
2
votes
2answers
71 views

File format where column names are repeated on each row

I have received a dataset in text file with the following format ...