Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

144 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Optimal practices to group data by Customer ID for churn prediction

Here's a quite common problem and I read a couple of questions/answers on it, however I still having my doubts about what are the best practices for grouping data by Customer ID for churn prediction. ...
0
votes
0answers
14 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
1answer
2k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
61 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
0
votes
0answers
21 views

Data Transformation Tips for xgboost's XGBClassifier

I have this X_train and test distribution for the 4 features 'X', 'Y', 'TX' and 'TY'. I realize the range of the distribution is widely varying .. Can you suggest a good way to clean/ transform that ...
0
votes
0answers
16 views

How do I clean the data for cluster analysis using sas miner

Hi I am looking to run some associative and clustering analyses using this dataset on kaggle. All the methods on Kaggle is in R and Python. Any ideas on how to clean and deal with the missing values? ...
0
votes
0answers
23 views

Averaging over Irregular Intervals

Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation? Specifically, suppose we are ...
0
votes
0answers
96 views

Where can I get an untokenized version of GLUE's SST-2 dataset?

On the GLUE faq, they say: Similarly, for SST, the data provided is already tokenized. We're working on obtaining a version that is not tokenized. Feel free to train on other distributions of ...
0
votes
0answers
27 views

How to handle a data set with large number (about 75%) of binary variables?

I am doing a research right now and want to classify (predict) churns of costumers using machine learning. My data set consists of about 500,000 observations with 20 variables: 15 are binary, 2 ...
0
votes
0answers
30 views

Transforming target from object array to integer array to use sparse_categorical_crossentropy for class prediction

I want to do a neural network to predict to which loan class does a borower pertains. There are 6 classes [ A, B, C, D, E, F]. I tried to get rid of the NAs and ...
0
votes
1answer
27 views

Grouping similar rows to detect duplicates

I have a dataset containing all the real estate ads for sale in the process of publishing a city: ...
0
votes
0answers
30 views

Model Checking in Multiple Imputation

Trying to understand how to incorporate Multivariate Imputation by Chained Equation (MICE) for handling missing data in python. I know there are a few libraries that can implement MICE on a dataset ...
0
votes
0answers
10 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
0
votes
0answers
28 views

Real-time Time-Series Error Correction

I have some sensors, each of which generates data points at mostly-regular intervals. So for each sensor I have a time series. I have to correct measurement errors from the sensors, replacing them ...
0
votes
3answers
137 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
0
votes
0answers
35 views

Intent Classification with imbalanced dataset

I am trying to train my ML model to classify 13 intents. Prior to training the model, the training data was sufficiently accurate. Here are the imbalanced training data sizes per class (class_num: ...
0
votes
0answers
37 views

Handling Multiple Classes in Categorical variables and Modeling help

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the ...
0
votes
0answers
20 views

Extract user input from a filled document form

I have some images of filled in documents with the same form scanned with different angle rotation, different luminosity, different noise etc, and also the image of empty form scanned correctly. My ...
0
votes
0answers
23 views

How do I export my data into .h5 format?

I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition. I need to prepare my own dataset in a format like this: ...
0
votes
0answers
38 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
149 views

Keras error in dimensions when predicting

I have trained a Keras LSTM model and was now trying to use it for predictions but for some reason in is giving a dimensions error I cannot find. I processed the data in the same way as the training ...
0
votes
1answer
122 views
0
votes
0answers
31 views

Creating artificial blurred image to train neural network for super resolution

I would like to create blurred images from images obtained from large pdf with the intention of creating a training data set. I would like to create the blur as shown in the uploaded picture. Which ...
0
votes
0answers
96 views

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file

CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar', 'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site- packages\tabula\tabula-1.0....
0
votes
0answers
366 views

Reading pdf file with all pages in Python error: CalledProcessError:

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file. ...
0
votes
0answers
29 views

Do I discard all my dependent variables as proved by chi-squared test of independence?

I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable ...
0
votes
0answers
26 views

Detect Typical Customer Mistakes in the Shopping Cart

I would like to ask your advice on solving this problem. Problem: There is an online store which sells furniture. There are millions of products on the store like furniture parts and furniture ...
0
votes
0answers
44 views

How to reduce / avoid false predictions with sklearn and MultinomialNB?

I'm using sklearn to predict product groups from product titles. That is working very well, if the titles are similar to the ones I labeled. Simplified example: ...
0
votes
2answers
486 views

Tips for checking data integrity / data sanity?

I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets. I am interested in ...
0
votes
2answers
83 views

Dataset Merging

I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
0
votes
1answer
419 views

Normalization and Outlier on Target variable which is continuous

I have doubt that should I perform outlier analysis and normalization even on target variable which is continuous ?
0
votes
1answer
38 views

How can I create a new column of binary values from my TfidfVectorizer sparse matrix?

I currently have a sparse matrix object of TfidfVectorizer which is of 1000 length. Right now it is displayed like this : ...
0
votes
1answer
115 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
341 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
1answer
103 views

Comparing data sets with different measurements

I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis. One piece of data (https://www.pandasecurity....
0
votes
1answer
295 views

Downsampling audio files for use in Machine Learning

I'm trying to use the work (Neural Networks) done in this repo: https://github.com/jtkim-kaist/VAD It says this: Note: To apply this toolkit to other speech data, the speech data should be ...
0
votes
1answer
282 views

Tool vs Python Script for Transforming Data in Mongo

We have a bunch of Mongo collections (data collected from APIs, web scraping, etc) that we need to transform to a cleaner data structure (standardized schema) on a monthly basis. Are there any good ...
0
votes
1answer
64 views

What are appropriate labels for age categorical labels?

I am converting some age data to categorical variables. What are some appropriate labels? Some people might take offense to using "Young", "Old" or "Millenial", etc. Is there a "standard" list of ...
-1
votes
2answers
2k views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
-1
votes
1answer
122 views
-1
votes
1answer
74 views

Text standardisation for manually entered data

I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies ...
-1
votes
1answer
109 views

Do I need equal number of bins for all attributes?

I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize ...
-1
votes
2answers
66 views

Creating a dataset for benchmarking of timeseries preprocessing capabilities

I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even ...
-1
votes
1answer
210 views

clean the data with sample selection?

I am using Python to do weather forecasting. Here is the original data: X.txt is the input, and y.txt is the output. https://drive.google.com/drive/folders/0BxxxnsxpOeLNUHRPdHdYUW9NYkk?usp=sharing ...

1 2
3