Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

K-means clustering for east west airline marketing data. Confused in choosing the optimal clusters

Ques: The file EastWestAirlines contains information on passengers who belong to an airline’s frequent flier program. For each passenger the data include information on their mileage history and on ...
0
votes
1answer
19 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
1
vote
0answers
12 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
1
vote
1answer
16 views

Dropping missing rows in two dataframes

I have two files : Test_data - contains the features of a dataset to find predictions for Submission_data - contains two columns : The index column for test data and another column for its ...
1
vote
1answer
37 views

Why we can't Remove features with missing values in Data Preprocessing [closed]

In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
0
votes
0answers
43 views

How to deal with categorical values in Data Analysis? [duplicate]

In machine learning problem, we are facing with categorical values like in the below code cell. So, Is OneHotEncoder feature is applicable for more then 10 or 20 unique values in perticular features. ...
1
vote
1answer
14 views

How we can Identify Specific Feature from a larger amount of Dataset?

In Machine Learning, we need to play with any kind of datasets. In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called ...
2
votes
1answer
26 views

Cleaning data with two fields mixed in the same column?

I am working on a template for a dashboard that is giving me some serious trouble. The format the data comes in is highly unstructured. In the image below, circled items are "Site Names", ...
1
vote
1answer
34 views

How to apply multiple filter in Data Frame? [closed]

How to implement multiple filters for checking data cell in a range ? Suppose, I have a list of numbers like, ...
4
votes
2answers
58 views

If there are no missing values in our training set, should we accommodate missing values in an unseen test set?

My training data has no missing values. I'm unsure whether or not I should fit say, imputation, on the training set so that I can accommodate possible missing values on the test set, because the test ...
0
votes
0answers
7 views

sklearn SimpleImputer using mean from different column groups?

I'm looking at the SimpleImputer, in particular in here, and I would like to do the imputation on different columns. My data has 3 different sample groups, and I would like to do ...
1
vote
0answers
13 views

Splitting Temporal Data

I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in ...
0
votes
1answer
15 views

How to expand abbrevations in text during preprocessing?

Im doing preprocessing on english text data. I have some domain specific abbreviations, for which i'm maintaining internal dictionary with key-value pairs. The problem i'm facing is the text has ...
1
vote
0answers
19 views

how to deal with columns that has different value in only 1 or 2 rows?

I have very high dimensional data. Almost 20% of the columns has different value in less than 1% of rows. All of these are binary columns and many columns has 0s filled in more than almost 98% of rows....
2
votes
1answer
33 views

Dealing with categorical variables in regression problems which method to use?

Usually if I have regression problem and my initial dataset contains categorical variables like : column 1: Math Science Science English I would convert this ...
3
votes
2answers
55 views

Treating missing data in categorical features

I have a dataset with one of the categorical columns having a considerable number of missing values. The interesting thing about this column is that it has values only for a particular category in &...
0
votes
1answer
35 views

Whether to replace NaN values in result column

I have a training dataset where we have to predict "Result" based on features "A", "B", "C" and "D" using machine learning. For a few rows, the "...
1
vote
1answer
18 views

A question on pipeline

I made a pipeline with standard scaler and k means .When I fit the pipeline to the training data, Does the standard scaler just fits or fits and transforms the training data?
2
votes
1answer
27 views

Does mini-batch gradient descent nullify the effect of stratification on the training data set?

In data pre-processing, stratified shuffle is used to ensure that the distribution of the original dataset is reflected in the training, test and validation dataset. Mini-batch gradient descent uses ...
2
votes
3answers
35 views

Combining Two CSV's in Jupyter Notebook

I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
1
vote
1answer
20 views

How would you encode missing pixels in image data?

I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it. Since the values are always positive, and ...
1
vote
1answer
69 views

How to prepare Audio-text data for speech recognition

I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I ...
1
vote
0answers
20 views

Is there a pandoc for data manipulation?

You know how pandocs converts markdown to HTML and pdf etc. Is there a pandocs for data manipulation? Like from SQL to pandas and SQL to dplyr etc?
0
votes
0answers
10 views

Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it?

I have Timeseries data to clean and I got the tip to use a kalman Filter. My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (...
0
votes
0answers
10 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
6
votes
3answers
719 views

Does label encoding an entire dataset cause data leakage?

I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the ...
0
votes
0answers
14 views

How does one decide on the right image quality filters for a training and testing set?

What image quality metrics do people use to choose appropriate quality (e.g. focus, orientation, obstruction, etc.) images for important for image classification models? I would imagine anything that ...
1
vote
1answer
51 views

How to access the data of the column on which some groupby operation has been carried out? [closed]

Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries ...
1
vote
1answer
21 views

Cleaning a certain feature to predict salary using Machine Learning

Info: I am working on a dataset, and i would like to create a model that would predict salary. Columns are as follows: ...
1
vote
1answer
18 views

How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument?

The only argument inside the fit function of SimpleImputer is: 'Age'. Yet the returned output worked on the 'Salary' column as well. That is what I am unable to understand. Here is my code (...
1
vote
1answer
31 views

Predicting equipment failure with time series alarm data

I am trying to predict machine failures based on alarm data. The situation: There is approximately 4000 machine failures per year. These are labelled poorly (it is entered manually and can have ...
0
votes
0answers
151 views

What are some standard or preferred battery of tests to perform to test for quality of my data before feeding to a ML algorithm?

we are designing a rules based engine to check the quality of the data before training our ML models. The data we have is time series data. We have about 3-4 features using which we have to make a ...
2
votes
1answer
19 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
0
votes
1answer
23 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
1
vote
1answer
36 views

Pandas - Sum of multiple specific columns [closed]

I created this script: ...
0
votes
0answers
16 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
1
vote
1answer
16 views

Reformatting data- Giving each value in a list it's own row while retaining the list's ID

I am looking to reformat some data. It currently looks like this: Using this as an example, the below is the format i'm trying to achieve: So each element in the list gets it's own row, but the ...
0
votes
1answer
19 views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
1
vote
1answer
45 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
0answers
44 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
0
votes
2answers
50 views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
1
vote
1answer
25 views

Train test split ERROR [closed]

So I was applying the random forest classifier model to a problem,however it showed this error ,even though the columns in X and Y of my dataset are equal. How can I resolve this? ...
2
votes
1answer
28 views

How to split data in R using dplyr if we want to have rows of the same group to belong to the same split?

In my current pipeline, I have sensed that there is data leakage. This is because the same person, though with slightly different values, is in both training and testing set. As a result, my model is ...
1
vote
0answers
26 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
0
votes
0answers
28 views

Best way to add border to rectangular images to train on StyleGAN2

TLDR: What is a best way to add borders to rectangular images with widely variable aspect ratios (from 0.5 to 2.0) to later use them as training data in StyleGAN2 with data resolution of 1024x1024 and ...
0
votes
0answers
20 views

How to convert google trend relative values into more “absolute” values?

Searching key words or terms on google trends provides a time series of relative search numbers. I want to be able to analyse these trends more accurately compared to general trends in the same ...
0
votes
1answer
17 views

Logistic Regression Multi-level Independent variables

im trying to study logistic regression, when i did the target variable with all features, i had the summary showing the p-values as usual, but one for the features has 60 level, another feature has 13 ...
0
votes
2answers
72 views

When should you remove outliers?

Let's say I've found some outliers in a column in my dataset and have decided to remove them. Should I do this before or after I split the dataset into train/test sets?
1
vote
0answers
26 views

Split-Half Reliability Python

I have a pandas dataframe of baseball stats that I have transposed that looks as follows: Each column represents a separate plate appearance for a player. Each player can (and usually is) represented ...
0
votes
1answer
28 views

CSVs: how to output missing data to make processing easier?

If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing? Some non-empty sentinel, like NA ...

1
2 3 4 5
11