Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
1answer
18 views

How to expand abbrevations in text during preprocessing?

Im doing preprocessing on english text data. I have some domain specific abbreviations, for which i'm maintaining internal dictionary with key-value pairs. The problem i'm facing is the text has ...
1
vote
0answers
21 views

how to deal with columns that has different value in only 1 or 2 rows?

I have very high dimensional data. Almost 20% of the columns has different value in less than 1% of rows. All of these are binary columns and many columns has 0s filled in more than almost 98% of rows....
2
votes
1answer
34 views

Dealing with categorical variables in regression problems which method to use?

Usually if I have regression problem and my initial dataset contains categorical variables like : column 1: Math Science Science English I would convert this ...
3
votes
2answers
64 views

Treating missing data in categorical features

I have a dataset with one of the categorical columns having a considerable number of missing values. The interesting thing about this column is that it has values only for a particular category in &...
0
votes
1answer
41 views

Whether to replace NaN values in result column

I have a training dataset where we have to predict "Result" based on features "A", "B", "C" and "D" using machine learning. For a few rows, the "...
1
vote
1answer
19 views

A question on pipeline

I made a pipeline with standard scaler and k means .When I fit the pipeline to the training data, Does the standard scaler just fits or fits and transforms the training data?
2
votes
1answer
50 views

Does mini-batch gradient descent nullify the effect of stratification on the training data set?

In data pre-processing, stratified shuffle is used to ensure that the distribution of the original dataset is reflected in the training, test and validation dataset. Mini-batch gradient descent uses ...
2
votes
3answers
142 views

Combining Two CSV's in Jupyter Notebook

I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
1
vote
1answer
26 views

How would you encode missing pixels in image data?

I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it. Since the values are always positive, and ...
1
vote
1answer
94 views

How to prepare Audio-text data for speech recognition

I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I ...
1
vote
0answers
20 views

Is there a pandoc for data manipulation?

You know how pandocs converts markdown to HTML and pdf etc. Is there a pandocs for data manipulation? Like from SQL to pandas and SQL to dplyr etc?
0
votes
0answers
13 views

Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it? [duplicate]

I have Timeseries data to clean and I got the tip to use a kalman Filter. My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (...
1
vote
0answers
19 views

How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like ...
6
votes
3answers
908 views

Does label encoding an entire dataset cause data leakage?

I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the ...
0
votes
0answers
14 views

How does one decide on the right image quality filters for a training and testing set?

What image quality metrics do people use to choose appropriate quality (e.g. focus, orientation, obstruction, etc.) images for important for image classification models? I would imagine anything that ...
1
vote
1answer
56 views

How to access the data of the column on which some groupby operation has been carried out? [closed]

Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries ...
1
vote
1answer
25 views

Cleaning a certain feature to predict salary using Machine Learning

Info: I am working on a dataset, and i would like to create a model that would predict salary. Columns are as follows: ...
1
vote
1answer
34 views

How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument?

The only argument inside the fit function of SimpleImputer is: 'Age'. Yet the returned output worked on the 'Salary' column as well. That is what I am unable to understand. Here is my code (...
1
vote
1answer
63 views

Predicting equipment failure with time series alarm data

I am trying to predict machine failures based on alarm data. The situation: There is approximately 4000 machine failures per year. These are labelled poorly (it is entered manually and can have ...
2
votes
1answer
28 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
0
votes
1answer
23 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
1
vote
1answer
40 views

Pandas - Sum of multiple specific columns [closed]

I created this script: ...
0
votes
0answers
30 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
1
vote
1answer
18 views

Reformatting data- Giving each value in a list it's own row while retaining the list's ID

I am looking to reformat some data. It currently looks like this: Using this as an example, the below is the format i'm trying to achieve: So each element in the list gets it's own row, but the ...
0
votes
1answer
193 views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
1
vote
1answer
51 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
0answers
55 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
0
votes
2answers
830 views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
1
vote
1answer
34 views

Train test split ERROR [closed]

So I was applying the random forest classifier model to a problem,however it showed this error ,even though the columns in X and Y of my dataset are equal. How can I resolve this? ...
2
votes
1answer
33 views

How to split data in R using dplyr if we want to have rows of the same group to belong to the same split?

In my current pipeline, I have sensed that there is data leakage. This is because the same person, though with slightly different values, is in both training and testing set. As a result, my model is ...
1
vote
0answers
27 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
0
votes
0answers
46 views

Best way to add border to rectangular images to train on StyleGAN2

TLDR: What is a best way to add borders to rectangular images with widely variable aspect ratios (from 0.5 to 2.0) to later use them as training data in StyleGAN2 with data resolution of 1024x1024 and ...
0
votes
0answers
75 views

How to convert google trend relative values into more “absolute” values?

Searching key words or terms on google trends provides a time series of relative search numbers. I want to be able to analyse these trends more accurately compared to general trends in the same ...
0
votes
1answer
23 views

Logistic Regression Multi-level Independent variables

im trying to study logistic regression, when i did the target variable with all features, i had the summary showing the p-values as usual, but one for the features has 60 level, another feature has 13 ...
0
votes
2answers
501 views

When should you remove outliers?

Let's say I've found some outliers in a column in my dataset and have decided to remove them. Should I do this before or after I split the dataset into train/test sets?
1
vote
0answers
59 views

Split-Half Reliability Python [closed]

I have a pandas dataframe of baseball stats that I have transposed that looks as follows: Each column represents a separate plate appearance for a player. Each player can (and usually is) represented ...
0
votes
1answer
30 views

CSVs: how to output missing data to make processing easier?

If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing? Some non-empty sentinel, like NA ...
0
votes
0answers
38 views

Selecting the optimal number of bins in KBinsDiscretizer?

I have Data Frame where are continuous values Features present. I want to bin these Features in category. I am using KBinsDescretizer for this. To find the optimal ...
2
votes
1answer
109 views

how does xgboost handle inf or -inf values?

I am using xgboost for binary classfication. I have infs and -infs in my data due to the fact I am calculating ratios from one col and and another e.g. ...
0
votes
1answer
18 views

Create new column with rounded values

I am trying to create a new column in this dataset. I would like to have rating to be 1, 2, 3, 4 or 5 instead of 1.25, 2.00, 3.75, ..., 5.00. Can you help me? The only solution I found is this ...
4
votes
1answer
24 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
4
votes
1answer
45 views

Data-preprocessing for Machine Learning model

I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the ...
1
vote
0answers
19 views

Excluding “mislabeled” examples in training set based on out of time data

This question is specifically regarding imperfect labels. I'd like to understand the theoretical and practical implications of removing examples from the training set based on information obtained ...
1
vote
0answers
31 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
1
vote
0answers
37 views

How to clean messy column and reshape data structure in pandas

I have a dataset that looks like this. There are 122 columns of coordinates excluding the date (Time) column and 7000 rows. What I want to do is split the lat and lon into two columns, and then ...
0
votes
0answers
14 views

Some doubts in sklearn.preprocessing.Normalizer ? Please explain in lucid manner without jagron

In sklearn.preprocessing.Normalizer it mentioned that “”Scaling inputs to unit norms is a common operation for text classification or clustering for instance. For instance the dot product of two l2-...
0
votes
0answers
21 views

Multiple Values for One Day

I have two questions. 1- I have weather data of 10 turbines and I know their collective production(Power).I also know maximum power a turbine can make. How can I forecast collective production if I ...
1
vote
1answer
17 views

Subdivide a numerical vector with a normal distribution

I have a numerical array of prices values. I'd like to do classification on this parameter, so I'd like to create a certain number of classes with the same granularity. I'd like to create a ...
2
votes
1answer
24 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
0
votes
0answers
11 views

How to identify corresponding record of a car from a semi-structured string?

I am trying to build an application that can take a record of a car from different websites, compare it to data i have in a CSV file and return me the matching row. Each website will present and ...

1
2
3 4 5
11