Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
1answer
18 views

Shaping columns (and column headers) into multi-index rows using pivot/pivot_table

Problem I am trying to shape a dataset which includes stacked dates/data as rows and half-hourly data as columns (Trading periods, eg. TP1, TP2), into a chronological half-hourly order with the ...
0
votes
1answer
19 views

How can I group by elements in a column in pandas? [closed]

I have a dataset that looks like this: I need to transform it so it looks like this: Meaning I need to show the balance by balance groups and time gaps. Now the time gaps are elements of a column. I ...
0
votes
1answer
21 views

How can I rename elements in a column in pandas? [closed]

I have a column named 'Segement' which contains elements named eiter Retail or Corporate, but it also containts sub segments such as Retail-Msf and mid. Corporate, etc, I have about 7000 of these and ...
0
votes
0answers
22 views

Convert a dataset to a binomial distribution in R [closed]

I have an excel file that has COVID-19 data for Indiana. Which can be found here: https://hub.mph.in.gov/dataset/covid-19-county-statistics. The data has the following columns: ZIP_CD (zipcode) ...
0
votes
0answers
15 views

How to filter data samples which do not improve classifier?

I have a text dataset with noisy labels and an unbalanced shape. There are various ways to find features which do not drive improvement in some metric, and help to prune those from the pipeline. I ...
0
votes
0answers
11 views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
-1
votes
1answer
12 views

Data wrangling dates

I have a feature with data creation dates. I have normalized them all to the same format and split them to 'day', 'month' and 'year' columns. But now I have a question. Should I apply normalization or ...
0
votes
0answers
11 views

Manually labeled data quality check methods?

In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation ...
1
vote
0answers
20 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
-2
votes
0answers
17 views

Big data analysis with Python [closed]

If I have a big dataset, say 1TB, and want to explore and analyze it in the same way I do in my Jupyter Notebook. Is there a way to work in a Jupyter Notebook with large data like this? How would you ...
0
votes
1answer
24 views

Calculate implicit rating from streaming behaviour for Recommendation Engine

I have a dataset containing some user streams data for particular videos like below: ...
0
votes
0answers
7 views

How to deal with a small dataset for image classification using CNN?

I have a dataset consisting of characters(lowercase and uppercase) and numbers, totalling about 62 classes. The data I have are about 45 images per class and no test data. The data is a subset of the ...
1
vote
1answer
10 views

Approach to labelling new data for training

Overview: Imagine an application that identifies cats and dogs from their phone camera. User's take a photo of their pet and it tells them if it is a dog or a cat. The data is then sent to the server. ...
0
votes
1answer
9 views

How to best visualize or capture time interval between lab measurements?

I have a table like as shown below ...
1
vote
0answers
12 views

How to take the keywords from the given dataset to train GPT-2 based chatbot?

I am working with a dataset that contains Questions on various Events conducted by a college and the corresponding answers for the queries. I am using this dataset to train a GPT-2 355M model to ...
1
vote
0answers
28 views

Should I drop Nan before before creating a scatter plot and computing the correlation matrix?

I have some NaN values in my data and I cannot replace it with median or something else. Now I have to check out the correlation, using scatter plot and Pearson correlation. Should I drop these NaNs ...
0
votes
0answers
14 views

Treating Null Degrees of an Angle

I have a dataset that measures the flight details of objects, based on what action was performed. It looks similar to below: ...
1
vote
0answers
7 views

Cleaning error text strings inside Categorical Variable Columns [closed]

I have a DataFrame (see below - the real dataset is about 10,000 rows, 26 columns) with errors like '?' or '&#-81;' where there should be a string corresponding to a value like a Premise or a ...
0
votes
0answers
22 views

Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?

I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution ...
2
votes
1answer
22 views

Manual Data Cleanup Tools

I am writing an ETL pipeline for geospatial data of the form place_name,address,longitude,latitude,id_linking_to_other_dataset As the last step in the pipeline, I ...
1
vote
0answers
16 views

Optimize Yahoo Finance Code for Analysis [closed]

I am trying to analyze a number of companies using financial data I gathered from Yahoo Finance. I am also using the yfinance API to get some more details about the company using functions. Since I am ...
0
votes
1answer
40 views

What is the best solution to replace NaN values?

I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that ...
1
vote
0answers
19 views

How to process categorical variable having lots of unique values in linear regression?

I have House Price dataset and I am using linear regression to predict the house price. while data preprocessing I found a variable called "Location" and it have around 342 unique value. For ...
0
votes
0answers
10 views

Why is my data not showing valid autocorrelation?

So i was following a kaggle notebook on weather prediction using ARIMA/SARIMA (here). I implemented the steps to look for autocorrelation on my dataset and found that the graph is in very strange ...
1
vote
1answer
58 views

Replace values in dataframe with another dataframes values based on condition [closed]

I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the ...
-1
votes
1answer
18 views

Increase accuracy in binary classification with ambiguous data

I am fairly new to Datascience and currently working on an assignment that requires me to do a binary classification on a set with about 9 parameters for X. I tried working on it using different ...
3
votes
2answers
110 views

Preprocessing: StandardScaler() Do we really need mean to be zero?

For instance, many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the l1 and l2 regularizers of linear models) assume that all ...
0
votes
1answer
32 views

Making (500,) array (500,20)

I have this array (500,): https://pastebin.pl/view/adeccfa1 Want to make it look like this one (500,20): https://pastebin.pl/view/6650ad6b How do I do it?
0
votes
1answer
24 views

How to remove irrelevant text data from a large dataset

I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed ...
0
votes
0answers
10 views

Do I remove the missing values before performing univariate and bivariate analysis? Is there a general rule?

I have read the answer to this question, but it doesn't quite answer my question whether there is a general rule for dealing with this situation. When performing EDA I find missing values in the ...
1
vote
0answers
14 views

Encoding Data for ML Modeling with Key Value Pair

Apologies in advance for the poor title. I didn't know exactly how to phrase what I want in a succinct manner so hopefully I can elaborate a bit more. I have a dataset where I have customer_id, ...
2
votes
1answer
54 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
0
votes
1answer
34 views

Data Cleaning Techniques - What to drop?

I have been struggling with this problem for a few weeks but it seems to be out of my league. I have a dataset of product sales over the course of 55 weeks. It contains information on the store ID, ...
2
votes
1answer
12 views

Architectures that take inputs of mixed sampling rates

Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time ...
1
vote
0answers
19 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
0
votes
0answers
29 views

Data calculation/normalisation /data summarisation

Clustering is done based on houses data and say suppose cluster 1 contains 5 houses. Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an ...
0
votes
0answers
9 views

Figuring out what's wrong with the box plot. Outliers?

What a 'box plot' of this kind has to say? that basically I have a lot of outliers and I should focus on data in proximity of zero?
3
votes
1answer
40 views

IterativeImputer - Returning -0 and other wierd results

I am using IterativeImputer to impute my dataset. ...
1
vote
1answer
49 views

NLP data cleaning and word tokenizing

I am new to NLP and have a dataset that has a bunch of (social media) messages on which I would like to try some methods like latent Dirichlet allocation (LDA). First, I need to clean the data of ...
0
votes
1answer
22 views

Cleaning NaNs with averages pre or post split? [duplicate]

I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode. Technically, the validation/ test data has never been seen before - so how could I include it in the ...
0
votes
0answers
18 views

how do i preprocess percentage data?

I am analyzing a problem where i have 5 diseases and the measure of effectiveness of a remedy in its 7 first applications. The data is organized as an Excel spreadsheet as follows: (The spreadsheet ...
1
vote
1answer
41 views

Is there a standardized way to do data analysis?

Is there a standard way to do data analysis? So for example, something like this: 1. Data mining 2. Data cleaning 3. xx 4. Data and result interpretations I ...
0
votes
1answer
32 views

The process of normalizing your data

I have a machine learning model to pick different outcome of each game for a college basketball game. The X values are: ...
0
votes
1answer
19 views

replace values based on Number of duplicate rows are occured

I have a dataframe ,that looks like this site Active 0 deals Active 1 deals Active 2 deals Active 3 discount Active 4 discount Active i ...
0
votes
0answers
10 views

In a dataset, how to detect simple relationship between two columns (datatype not restricted to numerical values)?

Suppose, I have a dataset with the following features: (FirstName, LastName, FullName, SchoolJoiningDate, SchoolGraduationDate) My aim is to validate the data before feeding it into the ML pipeline. ...
1
vote
0answers
41 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
1
vote
1answer
17 views

Document clustering to merge common labels

I am building a recommendation system and I have to clean up some of the labels that I have. For example of the data df['resolution_modified'].value_counts() Gives ...
0
votes
0answers
11 views

query category wise dara

I have stored the some products and there respective category in sql. Like this. Now I want fetch all product from table with respective category. output should be=>
2
votes
1answer
45 views

How modelling is affected by similar feature distributions across classes?

I have grouped my dataset according to labels (good and bad customers), in an attempt to test how each feature is distributed within. Along this, I found some feature has almost exact distribution in ...

1
2 3 4 5
12