Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

175 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1
vote
0answers
65 views

How to model this variable?

I'm doing some data prep on a dataset provided by a telecommunication company. There is a continuous variable that indicates how many months have passed since a customer renewed her contract. However, ...
1
vote
0answers
340 views

Scoring consistency within dataset

Suppose I am given a set of structured data. The data is known to be problematic, and I need to somehow "score" them on consistency. For example, I have the data as shown below: ...
1
vote
0answers
101 views

What's the best way to rank aggregate imdb rating data?

I have an average rating of all votes as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?: 1) multiply each average rating by the total ...
1
vote
0answers
70 views

What tools are available for semi-automated matching of dirty columnar data

Are there any automated or semi-automated tools for finding matching "similar" or data in two columnar data sets? The data I'm working with was collected (and handled) by different organizations. ...
0
votes
0answers
6 views

How to read multiple JSON files to a dataframe in R? can I get the code with the comments on it to understand it better?

I have multiple json files in a folder called "app users". These json files has NAN values as well in it.
0
votes
0answers
15 views

Handling repeating data from different individuals

I have a dataset that has some unique values but also includes information from multiple individuals that are repeating, meaning they are describing the same attributes and can have the same or ...
0
votes
0answers
26 views

Should I remove rows without target value?

I am new to data science so any help is appreciated. Thanks in advance. I have a data set with a few explanatory variables and a target column that contains the label of each row. However, some labels ...
0
votes
2answers
12 views

How do I remove float object in dataframe? Ex: "winter 2021" to "winter"

I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
0
votes
0answers
17 views

Scaling columns pandas DataFrame

For a dataset having different numeric columns, they usually have different range and distributions. As an example, I have used the Iris dataset. The distributions of it's 4 columns are shown: My ...
0
votes
0answers
29 views

Is it possible to derive anything useful from this piece of data?

Let's say you have online Profile A. Profile A is present on 3 websites: X, Y, Z. ...
0
votes
0answers
6 views

I am trying to sum up the "travel time in minute" for lane "GP" and grouping it under a single "Toll ID"

I am trying to sum up the "travel_time_minutes" for lane_type "GP" and grouping it under a single "Toll ID". I mean 0.11 + 0.50 + 1.21 for GP and retaining the Toll ID of ...
0
votes
0answers
13 views

Aggregating transactional data for customer segmentation

I have item-level transactional data where each row in the data represents a different item bought by a customer in a transaction (so if two different items were bought in the same transaction by the ...
0
votes
0answers
27 views

Identify the missing value in a table

How to compute the missing values x in the below table. which formula can be used to obtain the values of x. The values in the last row and last column are the sums of the rows and columns. What is ...
0
votes
1answer
45 views

Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
0
votes
0answers
42 views

Global models for time series prediction with series of different lengths

As I haven't found much on the topic, I wanted to start the discussion about local- (iteratively for every column) vs. global models (only one iteration per forecast horizon, ex ante). As this is a ...
0
votes
0answers
31 views

How can I balance sentence data for NLP tasks

I have been given a task to train the SVM model on conll2003 dataset for Named Entity "Identification" (That is I have to tag all tokens in "Statue of Liberty" as named entities ...
0
votes
0answers
15 views

fast detection of mislabelled data samples

I have about 20% mislabelled samples within my dataset and I am looking for "fast" ways to detect them, one way that occurred to me: step 1: build a classifier (with k fold) on the dataset ...
0
votes
0answers
11 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
0
votes
1answer
22 views

How to deal with data having 0 values in many columns?

I am trying to implement logistic regression but the dataset that I have have many columns with skewed data and most of them have 0 as values. I also the skewness of data for many columns its going ...
0
votes
0answers
8 views

How to trim the train, test and validate datasets for LSTM to avoid shape issues?

I am trying to get an LSTM to optimize for regression of stock returns, but I keep encountering shape errors during the fit() phase. Specified a list with shape [30,2] from a tensor with shape [1,2] ...
0
votes
1answer
25 views

How to improve regression neural network?

I am new to deep learning and data science and trying to increase my knowledge by working on some hackathons. Currently, the hackathon project I am working on has the task to predict the closing price ...
0
votes
0answers
9 views

How to deal with errors or inconsistencies in the training data?

There are inconsistant wrong labels and consistant errors in training data. For the former I tried MC-dropout and data Shapley. For the later I wonder if manual data curation is a requisite?
0
votes
0answers
23 views

How to mathematically quantify the quality of a corpus?

I am working on a text classification project. I have around 60,000 text samples of 40intents. By calculating the frequency of ...
0
votes
1answer
19 views

ML for data processing. What are the options?

Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity. This entails that in many cases ...
0
votes
0answers
27 views

EDA on multivariate time series data

I'm working with a medical dataset in a "long" format, meaning each row represents a timestamp and I have a column named "test" representing the test, an additional column ...
0
votes
0answers
17 views

formatting data from pandas into CNN-compatible array

I have a pandas dataframe of this form: time x y 0 a b 1 c d For my CNN model training, I want to slice the first 5 rows (from t=0 to t=4) and apply my custom function to label this sliced set, ...
0
votes
0answers
15 views

Time series classification - subject level analysis

I have a dataset with smartphone sensors data collection. There are 300 subjects, and the frequency of the sensors collected is different for different subjects. The shape of the data is (20k, 30) -&...
0
votes
0answers
30 views

A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
0
votes
0answers
11 views

Removing categories with low sample size

I have a categorical column with 4 unique labels: Left ventricular hypertrophy Normal rest ecg Wave abnormality Out of 831 rows, only 4 of them include wave abnormality. It is a really low sample ...
0
votes
0answers
7 views

How to clean/analyze relative time series data

I'm relatively new to data science, and I decided to stretch a bit and try a project with time series data. I downloaded the Human Activity Recognition from Continuous Ambient Sensor Data Data Set, ...
0
votes
1answer
16 views

Concatenating Data in two years

I have to use a Machine Learning Model to predict the Electricity consumption and carbon emission based on some buildings' features. (Area, year of construction ...) Here is the link to the data. The ...
0
votes
0answers
137 views

Com posso resolver esse erro: TypeError: a bytes-like object is required, not '_io.BufferedReader'

msg: erro: TypeError: a bytes-like object is required, not '_io.BufferedReader' My code: import pickle with open(b'ModelosParaTrader/ModeloEurUsd.pkcls', 'rb') as modelo: lr = pickle.loads(modelo) lr
0
votes
1answer
14 views

Advices to create a data set from dictionnary's screenshot

I have six thousand screenshots like the one below. I would like to create a date set to do some deep learning. My goal in the end is to create the next word prediction using only my own word data set....
0
votes
1answer
30 views

How can I calculate total days past due between billing events?

I am dealing with a dataframe with subscription events partitioned by username, subscription status, and relative timestamps. For each of the dates, there are changes in time when the subscription ...
0
votes
0answers
9 views

How to introduce a parameter for measuring change in data over time

In my project, I need to introduce a measure for 'movement' using a 3axis accelerometer (ADXL345). As sketched below: I thought about introducing some micro-changes, i.e. absolute change in ...
0
votes
0answers
11 views

Finding the wiggles pattern in the original dataset. (Wiggles appear after performing division by another dataset)

I have multiple measurements regarding scientific observations. The problem is that there is a subtle noise pattern caused by the instrument - the wiggles. These wiggles are invisible when looking at ...
0
votes
0answers
29 views

Synthetic data generation for Churn prediction

I am trying to predict which customers will churn in next three months for a Telecom service provider, and I don't have any data! I have to generate synthetic data and train some models. One thing is ,...
0
votes
0answers
28 views

Using Logical Functions in Tableau Prep for Address string editing

I have a data set with A LOT of inconsistent addresses. For example, in some cases you have STREET versus ST and "Fourth Ave" versus "4th Ave." I have been using Tableau Prep to ...
0
votes
0answers
18 views

How to filter data samples which do not improve classifier?

I have a text dataset with noisy labels and an unbalanced shape. There are various ways to find features which do not drive improvement in some metric, and help to prune those from the pipeline. I ...
0
votes
1answer
966 views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
0
votes
0answers
20 views

Manually labeled data quality check methods?

In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation ...
0
votes
0answers
20 views

How to deal with a small dataset for image classification using CNN?

I have a dataset consisting of characters(lowercase and uppercase) and numbers, totalling about 62 classes. The data I have are about 45 images per class and no test data. The data is a subset of the ...
0
votes
0answers
15 views

Treating Null Degrees of an Angle

I have a dataset that measures the flight details of objects, based on what action was performed. It looks similar to below: ...
0
votes
0answers
36 views

Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?

I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution ...
0
votes
1answer
64 views

What is the best solution to replace NaN values?

I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that ...
0
votes
0answers
11 views

Why is my data not showing valid autocorrelation?

So i was following a kaggle notebook on weather prediction using ARIMA/SARIMA (here). I implemented the steps to look for autocorrelation on my dataset and found that the graph is in very strange ...
0
votes
1answer
31 views

How to remove irrelevant text data from a large dataset

I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed ...
0
votes
0answers
26 views

Do I remove the missing values before performing univariate and bivariate analysis? Is there a general rule?

I have read the answer to this question, but it doesn't quite answer my question whether there is a general rule for dealing with this situation. When performing EDA I find missing values in the ...
0
votes
0answers
31 views

Data calculation/normalisation /data summarisation

Clustering is done based on houses data and say suppose cluster 1 contains 5 houses. Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an ...
0
votes
0answers
10 views

Figuring out what's wrong with the box plot. Outliers?

What a 'box plot' of this kind has to say? that basically I have a lot of outliers and I should focus on data in proximity of zero?