Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

143 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1
vote
0answers
35 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...
1
vote
0answers
22 views

Visulazing a specific value of a column with the corresponding values

I have recently started using data studio to make visualizations before diving into analysis and data prep! I have a column named NAME OF STORE that good different names and other columns like Date, ...
1
vote
0answers
114 views

How to encode H3 geohash in regression model

I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3. I'm struggling ...
1
vote
0answers
63 views

preparing time series data for building a rnn

I am preparing time series data for to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a ...
1
vote
0answers
12 views

How to a object type data which contains a string of unique id for feeding in my model?

I am having a dataset in which data type of unique id of a user is in object form. I need to convert it into Int for feeding this data into my model. here is first rows of my dataset. ...
1
vote
0answers
700 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using: ...
1
vote
0answers
402 views

Converting HTML tables to a data.frame in R

I am trying to convert an HTML table of daily data into a R data.frame for trend analysis. Using xml2 and rvest I am able to download the table into RStudio, but am struggling to A) create the ...
1
vote
0answers
46 views

How do I prepare data in which each output row depends on multiple input rows?

My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y. In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on ...
1
vote
0answers
39 views

Extracting the time duration for events from an event log

I have an event file (very similar to a log file) that logs event information that includes user data and time stamps. The events have an ordering for each user. So event E1 occurs before E2 that ...
1
vote
0answers
40 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
1
vote
0answers
21 views

How to prepare future data for training

Assume I have a large data of ecommerce website sessions with user id key (a user can have multiple sessions with random time between them). The data is on S3 in json gzipped format. On some sessions ...
1
vote
0answers
27 views

calculate days from a column to today in r

I have data set with a Date column, now i am trying to calculate duration in days from today to dates in date column with a new column named duration. I tried ...
1
vote
0answers
39 views

What are the best free reporting tools?

What are the best free Business Intelligence (Reporting) tools, that allows live updates on the reports and also sharing them to other users? Actually now I’m using the community edition of Pentaho ...
1
vote
0answers
24 views

What are the ways to merge data from different sources? Is there any software framework?

Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and ...
1
vote
0answers
66 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
1
vote
0answers
20 views

Linear Model: How to deal with predictors with a lot of missing/small values?

I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes. All these predictors have the same unit, as they are all ...
1
vote
0answers
529 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
1
vote
2answers
1k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
1
vote
1answer
449 views

Am I supposed to scale binary features alongside other numerical features?

When I am cleaning my data, I have some features which contain large numbers and some features that are binary. Should I scale the large features and then add the binary columns or just scale them all ...
1
vote
0answers
35 views

Find irregular bounding shape of 3D particle distribution

I have some 3D particle data e.g. (x0,y0,z0) (x1,y1,z1) (x2,y2,z2) ... I want to find the irregular bounding shape of the distribution. The image below shows an ...
1
vote
0answers
25 views

E-Learning related terminology

I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?
1
vote
0answers
85 views

Dimension reduction : binary description and regression

Let assume that I own a model agency (that is a far-fetched example), and I would like to create a tool for my HR department, that gives an estimate salary for any given profile. I have a huge ...
1
vote
0answers
366 views

Online noise removal techniques

There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman ...
1
vote
0answers
64 views

How to model this variable?

I'm doing some data prep on a dataset provided by a telecommunication company. There is a continuous variable that indicates how many months have passed since a customer renewed her contract. However, ...
1
vote
0answers
250 views

Scoring consistency within dataset

Suppose I am given a set of structured data. The data is known to be problematic, and I need to somehow "score" them on consistency. For example, I have the data as shown below: ...
1
vote
0answers
78 views

What's the best way to rank aggregate imdb rating data?

I have an average rating of all votes as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?: 1) multiply each average rating by the total ...
1
vote
0answers
69 views

What tools are available for semi-automated matching of dirty columnar data

Are there any automated or semi-automated tools for finding matching "similar" or data in two columnar data sets? The data I'm working with was collected (and handled) by different organizations. ...
1
vote
0answers
185 views

Natural Language text categorization using RapidMiner

I'm new to data mining, so this might sound like a very simple task to some. So, I work in reliability engineering in aviation & I have a set of data that is generated on a daily basis regarding ...
1
vote
2answers
227 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
0
votes
1answer
6 views

Dropping missing rows in two dataframes

I have two files : Test_data - contains the features of a dataset to find predictions for Submission_data - contains two columns : The index column for test data and another column for its ...
0
votes
0answers
7 views

sklearn SimpleImputer using mean from different column groups?

I'm looking at the SimpleImputer, in particular in here, and I would like to do the imputation on different columns. My data has 3 different sample groups, and I would like to do ...
0
votes
1answer
15 views

How to expand abbrevations in text during preprocessing?

Im doing preprocessing on english text data. I have some domain specific abbreviations, for which i'm maintaining internal dictionary with key-value pairs. The problem i'm facing is the text has ...
0
votes
1answer
35 views

Whether to replace NaN values in result column

I have a training dataset where we have to predict "Result" based on features "A", "B", "C" and "D" using machine learning. For a few rows, the "...
0
votes
0answers
10 views

Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it?

I have Timeseries data to clean and I got the tip to use a kalman Filter. My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (...
0
votes
0answers
10 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
0
votes
0answers
14 views

How does one decide on the right image quality filters for a training and testing set?

What image quality metrics do people use to choose appropriate quality (e.g. focus, orientation, obstruction, etc.) images for important for image classification models? I would imagine anything that ...
0
votes
0answers
151 views

What are some standard or preferred battery of tests to perform to test for quality of my data before feeding to a ML algorithm?

we are designing a rules based engine to check the quality of the data before training our ML models. The data we have is time series data. We have about 3-4 features using which we have to make a ...
0
votes
1answer
23 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
0
votes
0answers
16 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
0
votes
0answers
44 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
0
votes
0answers
28 views

Best way to add border to rectangular images to train on StyleGAN2

TLDR: What is a best way to add borders to rectangular images with widely variable aspect ratios (from 0.5 to 2.0) to later use them as training data in StyleGAN2 with data resolution of 1024x1024 and ...
0
votes
0answers
20 views

How to convert google trend relative values into more “absolute” values?

Searching key words or terms on google trends provides a time series of relative search numbers. I want to be able to analyse these trends more accurately compared to general trends in the same ...
0
votes
1answer
17 views

Logistic Regression Multi-level Independent variables

im trying to study logistic regression, when i did the target variable with all features, i had the summary showing the p-values as usual, but one for the features has 60 level, another feature has 13 ...
0
votes
0answers
18 views

Selecting the optimal number of bins in KBinsDiscretizer?

I have Data Frame where are continuous values Features present. I want to bin these Features in category. I am using KBinsDescretizer for this. To find the optimal ...
0
votes
1answer
45 views

how does xgboost handle inf or -inf values?

all, i am using xgboost for binary classfication. I have infs and -infs in my data due to the fact i am calcaulting ratios from one col and and another e.g. df[col1]/df[col2] , since i have zeros ...
0
votes
0answers
13 views

Some doubts in sklearn.preprocessing.Normalizer ? Please explain in lucid manner without jagron

In sklearn.preprocessing.Normalizer it mentioned that “”Scaling inputs to unit norms is a common operation for text classification or clustering for instance. For instance the dot product of two l2-...
0
votes
0answers
20 views

Multiple Values for One Day

I have two questions. 1- I have weather data of 10 turbines and I know their collective production(Power).I also know maximum power a turbine can make. How can I forecast collective production if I ...
0
votes
0answers
11 views

How to identify corresponding record of a car from a semi-structured string?

I am trying to build an application that can take a record of a car from different websites, compare it to data i have in a CSV file and return me the matching row. Each website will present and ...
0
votes
1answer
42 views

How do I iterate over my images in dataset?

I am building an autoencoder with help from this site. There I was trying to build an autoencoder for my own custom data. My images are stored in a folder IMG and have names like ...
0
votes
0answers
17 views

How to use data provided in EEM20 forecasting competition?

I am new to competitions. I have gone through few Kaggle competitions and most of the time, they provide train data, test data and other supplementary data. Recently, I came across a new competition (...