Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0 votes
1 answer
7 views

How to convert city name prefix abbreviations?

Is there any standard tool or library or list for expanding town name abbreviations? For example "MT HOLLY" -> "MOUNT HOLLY" or "ST MICHAELS" -> "SAINT ...
user avatar
  • 111
1 vote
2 answers
20 views

Converting voting district GeoID to approximate zip code?

How do you convert a "GEOID", representing a US voting district, to an approximate ZIP code? I've extracted the GEOIDs from the JSON published by the New York Times of the results of the ...
user avatar
  • 111
0 votes
0 answers
5 views

Asynchronous vs synchronous polling when collecting time-series data from various sensors?

I'm leading a project in my small company where we are performing data collection on some water heating equipment we supply. I will be using a Raspberry Pi with Python scripts to aggregate data from a ...
user avatar
-1 votes
0 answers
18 views

Calculation of data sparsity in data warehouse

suppose the data warehouse of an organisation contains 4000 customers 3000 products and only 500 sold per day to different customers? calculate the sparsity of data warehouse if we kept data for 10 ...
user avatar
  • 1
0 votes
0 answers
11 views

Dealing with survey data across several years

I am working on a dataset and a major part of this dataset is the survey results in the past 5 years. The survey data includes 8 different perspectives such as self-rating, satisfaction rate, etc. All ...
user avatar
  • 1
0 votes
0 answers
14 views

How do I handle known trend inconsistency in data?

I have a dataset of house prices where during a certain period there was recession in the housing market (late 2008 to early 2012) and prices fell. The problem is the data during this period makes up ...
user avatar
0 votes
2 answers
20 views

Joining on columns with duplicate values - clean before merging or after merging?

I am joining on two data sets on a column which has duplicated values in both datasets. Is it better practice to remove the duplicates and make the values I am joining on a primary key in both ...
user avatar
1 vote
1 answer
32 views

Techniques for classification AI with sparse labels

I want to create an AI to classify images with a large set of labels (1000+ labels). However, the labels in the data set are correct but each image is not fully labelled. This means that each image's ...
user avatar
  • 11
0 votes
0 answers
15 views

Changing a 0-1 column datatype from int64 to uint8 such as in pandas.get_dummies()

Is it advisable to change the datatype int64 of a 0-1's column to uint8 such as ...
user avatar
0 votes
0 answers
25 views

Where can I view the ImageNet classes as a hierarchy on WordNet?

I always find a list of classes on Github that represent the synset ID and name of each Imagenet class label. I need to view the WordNet hierarchy of ImageNet as a tree so I can prune some classes ...
user avatar
  • 113
0 votes
0 answers
9 views

Approach to check multicollinearity for data with categorical and continuous features?

I would like to know what should be the best approach to check for multicollinearity if my data has categorical and continuous variables like:- Age, Income, Department(more than 2 category), Gender(2 ...
user avatar
0 votes
0 answers
11 views

How to normalize Date feature i.e Year, month, day etc for Deep learning or Classical machine learning models?

We know that it is best practice to normalize our data before passing it into any model. Normalizing continuous valued feature is trivial, for categorical features there is no issue of normalizing (do ...
user avatar
0 votes
0 answers
8 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
user avatar
  • 21
0 votes
1 answer
23 views

what degree of fredom one should use while calculating standard diviation for standardizing data

I am writing a function to standardize the data and I found out that we can choose either ddof = 0 or ddof = 1, so I got confused that which one to choose and why? Does this make any difference?
user avatar
0 votes
0 answers
5 views

How to disentangle non-mutually exclusive items coded in same question?

I have to work with a dataset where people ostensibly had the option to check several options to a question (eg "check all that apply"). But in the data, all of the options when selected are ...
user avatar
  • 1
0 votes
1 answer
18 views

Performing EDA on a dataset with missing features

I'm new to DS. I want to perform EDA on such dataset, where these are the missing features stats of my train and test sets: train: Test_0 0 Test_1 31 Test_2 0 Test_3 141 ...
user avatar
0 votes
0 answers
9 views

Scaling and handling highly correlated features in tabular data for regression

I am working on a regression problem trying to predict a target variable with seven predictor variables. I have a tabular dataset of 1400 rows. Before delving into the machine learning to build a ...
user avatar
  • 101
1 vote
1 answer
52 views

Data preprocessing methods

Data Cleaning Data Imbalance solving (Classification) Data Smoothing (decreasing noise) Creating-deleting features from original data Data Transformation (Box-cox,Log Transform) Making Dataset ...
user avatar
  • 45
0 votes
0 answers
35 views

R code making 1 column into multiple columns with their unique ID

Currently stuck on a data wrangling question in R. So far I've tried variations of this code using tidyverse package, columns 5 and 6 here were the rating and the user: ...
user avatar
0 votes
0 answers
13 views

Identify the existence of a wall like structure in a given int array

The idea is that I will query an API endpoint which will return me an array consisting of a price value and a quantity value [price, quantity]. In this dataset ...
user avatar
  • 1
0 votes
1 answer
34 views

How can we predict a value after several rows of data?

I have a regression problem in which for each week I have several rows (variable between rows i.e 1 week might have 1800 rows and other might have 5000 rows). My target is to predict a value at end of ...
user avatar
  • 1
0 votes
0 answers
11 views

Has anyone joined FHIR resources?

I have FHIR-formatted data (Fast Healthcare Interoperability Resources), that I've converted to CSV using the npm package ndjson-to-csv, and on first glance, this seems useable for ML experimentation, ...
user avatar
0 votes
1 answer
33 views

Data preparation for Machine Learning

I am new to machine learning. I have task at hand to predict outcome of a "process" for a set of inputs. The problem is that the "process" does not produce the exact same output ...
user avatar
1 vote
1 answer
38 views

Data Wrangling and data cleaning

I found some information about Data Wrangling and they say different things. In this one, they say data cleaning is a subcategory of data wrangling link In this PDF, data wrangling is EDA and model ...
user avatar
  • 459
0 votes
1 answer
14 views

What to do with missing origin city values

Hello fellow data scientists. I am new in this field, and I face to a problem, for what I need an advice. So I have data, where one column is product ID, and another which say from which city it ...
user avatar
0 votes
1 answer
53 views

Find dips in the time series data

I need help to remove dips(trough) from the signals. The red circle in the image indicates the dip and yellow circles indicates contributing points. Means there’re multiple points in the dip. I tried ...
user avatar
  • 101
0 votes
0 answers
13 views

Efficient way to compare one record to millions of rows

We have a production table that contains a bucket of customer data. A customer could be the same customer/person at location A and at location B. They are different by how the name is spelled, address ...
user avatar
0 votes
1 answer
27 views

Is normalization needed for TargetEncoded Variables?

Basically the title. If I encode the address of people (the cities they live in) with a target encoder, do I still need to normalize that column? Of course, the capital is going to have more citizens ...
user avatar
  • 1,248
0 votes
0 answers
16 views

Is There Techniques for creating synthetic Data for Regression Problem i tried SMOTE and its variant but these are for classification problem

This is my data "Volume" is my Target variable and all other are Independent variables i just applied labelencoder on Area_categ , wind_direction_labelencod and on current _label_encode and ...
user avatar
0 votes
0 answers
10 views

Industry analysis - multiple industries

I am trying to run logistic regression on marketing leads and use industry as a predictor of whether the lead converts (1/0). Often, when I enrich data from websites like crunchbase the associated ...
user avatar
0 votes
1 answer
35 views

Multivariate data preprocessing

I am trying to understand how multivariate data preprocessing works but there are some questions in my mind. For example, I can do data smoothing, transformation (box-cox, differentiation), noise ...
user avatar
1 vote
1 answer
139 views

how many rows does pandas' interpolate consider? [closed]

How does pandas' DataFrame.interpolate() work in relation to the amount of rows it considers: is it just the row before the NaNs and the row right after? Or is it the whole DataFrame (how does that ...
user avatar
0 votes
0 answers
6 views

Best way to get independent rows in your data set?

I have contract data which has some repeats of accounts so some of the rows are dependent on one another. How can I deal with this? I was thinking to put each account's data into a single row, but ...
user avatar
  • 1
1 vote
1 answer
37 views

Flattening column data with split then merging df with Pandas

Using names = df['Name and Location'].str.split(',', expand=True) I am able to split this dense data at delimiters like colons. I'm stuck on how to recombine the ...
user avatar
0 votes
1 answer
118 views

Should i always transform data to normal distribution?

I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
user avatar
  • 45
1 vote
1 answer
12 views

How to perform linear regression on a parameter that represents state/configuration of a machinery in a production process?

I am trying to perform linear regression on a manufacturing process in order to determine the influencing parameters on a particular product. The thing is there are several production parameters, and ...
user avatar
0 votes
0 answers
6 views

Is data clustering a good fit for security baselining in a disordered, widely varying dataset? Or is there a better alternative?

I want to see what value I can process from some security logs. What's typically taught in introductions to data clustering involves fairly 'normal' data where every row relates fairly well to the ...
user avatar
  • 101
2 votes
1 answer
25 views

Can I create a new target value based on the average target value of same data points for regression?

I am trying to predict profit of retail stores. The orginal dataframe looks like this: Store No feature A feature B year profit A 1 2 2016 20000 A 1 2 2017 40000 B 4 3 2017 50000 B 4 3 2018 40000 ...
user avatar
0 votes
0 answers
8 views

What exactly is the "currency" dimension in data profiling?

...
user avatar
  • 133
1 vote
0 answers
28 views

Is it possible to use structured(tabular) data as a reinforcement learning environment?

I want to do an RL project in which the agent will learn to drop duplicates in a tabular data. But I couldn't find any examples of RL being used that way - checked the RL based recommendation systems ...
user avatar
  • 11
0 votes
0 answers
17 views

Group a spark dataframe by a starting event to an ending event

Given a series of events (with datetime) such as: failed, failed, passed, failed, passed, passed I want to retrieve the time from when it first "failed" to when it first "passed," ...
user avatar
  • 101
0 votes
1 answer
22 views

if feature has 4 unique values ( number of products: 1 ,2,3,4 ) so should i treat that as categorical or discreate variable?

I am using bank churn data (https://www.kaggle.com/kmalit/bank-customer-churn-prediction/data) there is a column in data called NumOfProducts that has 4 unique values so should I treat that as a ...
user avatar
0 votes
1 answer
12 views

Replacing missing missing view count when photo

I'm currently analyzing a dataset of posts on Facebook. Some are videos and others are photos. One of the features is view_count which has missing values every time the post is a photo. How can I ...
user avatar
  • 93
0 votes
0 answers
19 views

How to manage sampling bias between training data and real-world data?

I'm currently working on a binary classification problem. My training dataset is rather small with only 1000 elements. (I don't know if it is relevant : my problem is similar to the "spam ...
user avatar
  • 1
1 vote
1 answer
17 views

Merging 2 datasets

In the given task, they've provides us with 2 datasets namely the test & train dataset. So, I was wondering if we could merge those 2 datasets into 1 data-frame and delete the duplicates. Would ...
user avatar
0 votes
0 answers
19 views

Cleaning data with conflicting data points

I have a dataset from a legacy system that is entered by various entities (internal technicians, internal billing clerks, customers, and some system-derived entries). The goal is to deliver cost per ...
user avatar
0 votes
1 answer
25 views

What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
user avatar
1 vote
1 answer
49 views

Why is an ML algorithm performing better with correlated features, than the one with uncorrelated ones?

I have a dataset with all numerical values. Since the features were not many, I created more by multiplying pairs of each other. This created some highly correlated features, as expected. Now, I ...
user avatar
0 votes
0 answers
19 views

Handling multiple object types in a large dataset to build a better ML model

I have a dataset with multiple object data_types. From the dataset, most of them were made into categorical features (Using One-hot encoding as well). But few of them have object data types with 6M ...
user avatar
0 votes
2 answers
185 views

How to remove outliers properly?

I was wondering what is the best practice for removing outliers from data. Plotting a boxplot for each feature (column of the dataset) and removing data that fall outside the whiskers seems like a ...
user avatar
  • 73

1
2 3 4 5
14