Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

0
votes
1answer
4 views

Am I justified in dropping this independent variable?

I'm currently doing churn prediction in R and during EDA, I discovered that a variable, say gender, has 1720 males who don't churn, and 280 males who do. Also, it has 864 females who don't churn, and ...
1
vote
1answer
7 views

Merging dataframes in Pandas is taking a surprisingly long time

I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
0
votes
1answer
9 views

how do i separate individual data from a single string

A lot of data like **current address, current city ...**etc are missing in the majority of strings. I know how to preprocess data but the noob in me is unable to categorize the data from a single ...
0
votes
1answer
20 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
0
votes
0answers
6 views

Redundancy measure of CSV files [closed]

I would like to have some quantity measure of how much redundant data we store in our CSV files - which are the outcome of testing process. The CSV files looks as below: And this goes for thousands ...
0
votes
1answer
32 views

Dimension reduction for data with categorical features [closed]

I am trying to reduce the dimensionality of the dataset. My data contains a large number of categorical features which are creating problems with the dimensionality reduction techniques I am using (...
0
votes
0answers
6 views

How should I create attributes in my dataset if any permutation of the values yield the same result?

I am trying to make a movie rating predicting system. My approach is that the team (actors, directors, writers, producers, etc) is the determining factor. There can be more than one of each of those ...
0
votes
0answers
12 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
0answers
28 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
0answers
10 views

Will mean-value imputation have the same effect in these two cases before normalization?

Will mean-value imputation have the same effect, if performed before normalization, on the distribution of the normalized values for those values that were originally not missing, for min-max ...
0
votes
0answers
9 views

Which is the best way to generate synthetic data for data science development

I have a data pipeline in place for streaming and batch data, which collects data from various sources and land it into hadoop's hdfs storage. I want to generate synthetic data with the same schema of ...
0
votes
0answers
5 views

operations on multiple entries in one column based on conditions meet from multiple column entries

I have a data set where I need to find the difference between the first time entry and last time entry based on the following conditions: Day column entries are equal BusOne[0] = BusOne1 ... BusOne[n]...
3
votes
3answers
548 views

Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I ...
0
votes
0answers
10 views

Excessive Data checking

This probably too opinion-based. But, I am still a newbie to data science, and I often do not know how far is far enough when it comes to my analyses or ensuring data quality. A silly example: ...
0
votes
1answer
12 views

Converting string_id to number_id

I have column with movie ids like this: tt0984332 tt0984332 tt0847742 ttnanana1 I need to convert in to numbers that can be inserted into neural network as features, like this: 0 0 1 2 How can I do ...
0
votes
0answers
14 views

How to read a LISP-readable form data into python?

I have been trying to read this data into python. I tried various techniques but I am not able to organise it in a format that can be used for training a model. Has anybody got any idea how to read ...
0
votes
0answers
6 views

How to smoothen array of samples?

I am implementing a Genetic Algorithm library. However, on Roulette Wheel Selection algorithm sometimes one or two chromosomes dominate he roulette wheel with their relatively large fitness scores. I ...
0
votes
1answer
47 views

Regression in Python with many NaN values spread across all columns

I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank ...
1
vote
1answer
48 views

One Hot Encoding of Age

My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age. Statistically, it's a conditional probability. Example: ...
1
vote
0answers
55 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
0
votes
1answer
21 views

Cleaning the univariate dataset with high noise

At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension. For example: ...
1
vote
1answer
22 views

Wave form analysis ML algorithm [closed]

I am trying to determine a person's emotions from their speech. This immediately rings Machine Learning bells and the first step in any ML problem is getting and processing data. My first question is, ...
0
votes
2answers
39 views

Building predictive model with low correlated data [closed]

I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
-1
votes
0answers
19 views

Is there anything like an GUI/IDE for data science - automatically generating graphs and statistics etc?

In software development, IDE's (interactive development environments) help programmers by automatically generating a lot of useful stuff while debugging - variable values, call stacks, call graphs, ...
0
votes
1answer
12 views

How to do multivariate survival analysis on dataset having only categorical variables

I have a dataset where I have around 50 independent variables used to run survival analysis on the target variables. But out of these 50 variables, 46 variables are categorical variables i.e. having ...
1
vote
1answer
37 views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
0
votes
1answer
15 views

Does discretization of continuous features also lose information about distance?

During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way. But doesn't it also lose information about distances of features? For example, if we ...
1
vote
0answers
13 views

Converting HTML tables to a data.frame in R

I am trying to convert an HTML table of daily data into a R data.frame for trend analysis. Using xml2 and rvest I am able to download the table into RStudio, but am struggling to A) create the ...
1
vote
0answers
19 views

How do I prepare data in which each output row depends on multiple input rows?

My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y. In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on ...
1
vote
3answers
140 views

Data Cleaning without pandas [closed]

How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the ...
-1
votes
0answers
19 views

Need help to speed up code for stitching together badly formatted log files

The objective: I needed to stitch together a large number of badly formatted log files. I [painfully] developed a script for it in R, one that successfully dealt with the various flaws in the dataset. ...
1
vote
2answers
223 views

Pandas Conditional Fill NaN Forward/Backward

Updated 22 Oct. 2018: I have the following dataset: ...
0
votes
0answers
25 views

Handle outliers, Losing many data by removing natural outliers

I have 2 skewed features, here is the summary of one of the features ...
0
votes
2answers
51 views

Delete rows of a column using another column statistics in R [closed]

I have a dataframe with the following columns: If a year (see "year" column) has less than 10 weeks reported, I would like to delete it from the dataframe. The column "weeks_reporting" gives the ...
0
votes
3answers
489 views

How to access substrings in pandas column and store it into new columns?

I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like ...
0
votes
0answers
20 views

I want to create a Data cleaning module for English language to clean certain Emails. How do i begin?

Here is an Example of an Email in my .xlsx file. End goal is to perform text summarization and Topic modelling of the various Email's present in the Excel file.
0
votes
3answers
435 views

How to replace a part string value of a column using another column

How to replace a part string value of a column using another column. My DataSet here is : ...
0
votes
1answer
36 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
1
vote
0answers
25 views

Extracting the time duration for events from an event log

I have an event file (very similar to a log file) that logs event information that includes user data and time stamps. The events have an ordering for each user. So event E1 occurs before E2 that ...
1
vote
1answer
270 views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
0
votes
0answers
14 views

What keywords should I be looking for when verifying web-mined data?

I am scraping websites using the scrapy tool. The websites are all for restaurants and similar places. I have the name, latitude, and longitude information for them from OpenStreetMap. Sometimes ...
2
votes
2answers
102 views

How to get a count the number of observations for each year with a Pandas datetime column?

I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year. This is the column: ...
1
vote
1answer
36 views

What methods can be used to detect duplicacy in image dataset?

I want to remove duplicate images from a dataset of 50Million images. What is the best method to detect all the duplicates? Do you think one shot learning is good for this?
2
votes
1answer
27 views

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
1
vote
0answers
37 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
2
votes
0answers
31 views

Data formatting or Data preparation

...
0
votes
1answer
326 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
3
votes
1answer
55 views

Can someone please explain what this sample function is upto?

So there is a function in Dino_Name_Generator at Deeplearning.ai notebook ...
1
vote
6answers
103 views

How to statistically prove that a column in a dataframe is not needed

I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
1
vote
3answers
45 views

Data aggregation and split train test samples

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e....