Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument?

The only argument inside the fit function of SimpleImputer is: 'Age'. Yet the returned output worked on the 'Salary' column as well. That is what I am unable to understand. Here is my code (...
1
vote
1answer
22 views

Predicting equipment failure with time series alarm data

I am trying to predict machine failures based on alarm data. The situation: There is approximately 4000 machine failures per year. These are labelled poorly (it is entered manually and can have ...
0
votes
0answers
79 views

What are some standard or preferred battery of tests to perform to test for quality of my data before feeding to a ML algorithm?

we are designing a rules based engine to check the quality of the data before training our ML models. The data we have is time series data. We have about 3-4 features using which we have to make a ...
2
votes
1answer
16 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
0
votes
1answer
19 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
1
vote
1answer
30 views

Pandas - Sum of multiple specific columns [closed]

I created this script: ...
0
votes
0answers
10 views

DBSCAN vs RANSAC for outlier detection

As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?
1
vote
1answer
15 views

Reformatting data- Giving each value in a list it's own row while retaining the list's ID

I am looking to reformat some data. It currently looks like this: Using this as an example, the below is the format i'm trying to achieve: So each element in the list gets it's own row, but the ...
0
votes
1answer
14 views

Creating a new Dataframe with specific row numbers from another

I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers ...
1
vote
1answer
32 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
0answers
36 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
0
votes
2answers
43 views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
1
vote
1answer
21 views

Train test split ERROR [closed]

So I was applying the random forest classifier model to a problem,however it showed this error ,even though the columns in X and Y of my dataset are equal. How can I resolve this? ...
2
votes
1answer
25 views

How to split data in R using dplyr if we want to have rows of the same group to belong to the same split?

In my current pipeline, I have sensed that there is data leakage. This is because the same person, though with slightly different values, is in both training and testing set. As a result, my model is ...
1
vote
0answers
17 views

Custom DataFrame format for exporting to excel sheets

I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image ...
0
votes
0answers
17 views

Best way to add border to rectangular images to train on StyleGAN2

TLDR: What is a best way to add borders to rectangular images with widely variable aspect ratios (from 0.5 to 2.0) to later use them as training data in StyleGAN2 with data resolution of 1024x1024 and ...
0
votes
0answers
19 views

How to convert google trend relative values into more “absolute” values?

Searching key words or terms on google trends provides a time series of relative search numbers. I want to be able to analyse these trends more accurately compared to general trends in the same ...
0
votes
1answer
16 views

Logistic Regression Multi-level Independent variables

im trying to study logistic regression, when i did the target variable with all features, i had the summary showing the p-values as usual, but one for the features has 60 level, another feature has 13 ...
0
votes
2answers
33 views

When should you remove outliers?

Let's say I've found some outliers in a column in my dataset and have decided to remove them. Should I do this before or after I split the dataset into train/test sets?
1
vote
0answers
16 views

Split-Half Reliability Python

I have a pandas dataframe of baseball stats that I have transposed that looks as follows: Each column represents a separate plate appearance for a player. Each player can (and usually is) represented ...
0
votes
1answer
25 views

CSVs: how to output missing data to make processing easier?

If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing? Some non-empty sentinel, like NA ...
0
votes
0answers
16 views

Selecting the optimal number of bins in KBinsDiscretizer?

I have Data Frame where are continuous values Features present. I want to bin these Features in category. I am using KBinsDescretizer for this. To find the optimal ...
0
votes
1answer
30 views

how does xgboost handle inf or -inf values?

all, i am using xgboost for binary classfication. I have infs and -infs in my data due to the fact i am calcaulting ratios from one col and and another e.g. df[col1]/df[col2] , since i have zeros ...
0
votes
1answer
16 views

Create new column with rounded values

I am trying to create a new column in this dataset. I would like to have rating to be 1, 2, 3, 4 or 5 instead of 1.25, 2.00, 3.75, ..., 5.00. Can you help me? The only solution I found is this ...
3
votes
1answer
18 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
2
votes
1answer
31 views

Data-preprocessing for Machine Learning model

I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the ...
1
vote
0answers
18 views

Excluding “mislabeled” examples in training set based on out of time data

This question is specifically regarding imperfect labels. I'd like to understand the theoretical and practical implications of removing examples from the training set based on information obtained ...
1
vote
0answers
13 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
0
votes
0answers
22 views

How to clean messy column and reshape data structure in pandas

I have a dataset that looks like this. There are 122 columns of coordinates excluding the date (Time) column and 7000 rows. What I want to do is split the lat and lon into two columns, and then ...
0
votes
0answers
13 views

Some doubts in sklearn.preprocessing.Normalizer ? Please explain in lucid manner without jagron

In sklearn.preprocessing.Normalizer it mentioned that “”Scaling inputs to unit norms is a common operation for text classification or clustering for instance. For instance the dot product of two l2-...
0
votes
0answers
19 views

Multiple Values for One Day

I have two questions. 1- I have weather data of 10 turbines and I know their collective production(Power).I also know maximum power a turbine can make. How can I forecast collective production if I ...
0
votes
1answer
14 views

Subdivide a numerical vector with a normal distribution

I have a numerical array of prices values. I'd like to do classification on this parameter, so I'd like to create a certain number of classes with the same granularity. I'd like to create a ...
1
vote
1answer
20 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
0
votes
0answers
10 views

How to identify corresponding record of a car from a semi-structured string?

I am trying to build an application that can take a record of a car from different websites, compare it to data i have in a CSV file and return me the matching row. Each website will present and ...
0
votes
1answer
26 views

How do I iterate over my images in dataset?

I am building an autoencoder with help from this site. There I was trying to build an autoencoder for my own custom data. My images are stored in a folder IMG and have names like ...
0
votes
0answers
17 views

How to use data provided in EEM20 forecasting competition?

I am new to competitions. I have gone through few Kaggle competitions and most of the time, they provide train data, test data and other supplementary data. Recently, I came across a new competition (...
0
votes
0answers
19 views

Flow System - Analysing Measurement Errors

I have a closed system with multiple flow measurement points as shown; where in theory (A1 + A2 + A3) = (B1 + B2 + B3) = C = (D1 + D2) and D1 = (E1 + E2 + E3 + E4) D2 = (F1 + F2 + F3 + F4) Some of ...
1
vote
1answer
45 views

How to fix spelling mistakes in data?

I have an input data file which contains list of drug names. I have more than 1000 unique drug names. However, the drug names has spelling mistakes and space character issues. For ex: we have ...
1
vote
1answer
47 views

Dummy Variable Trap

In my course about machine learning I'm studying multiple linear regression and we talked about dummy variable trap. I have a data set which contains country, height, weight, gender of every person ...
1
vote
0answers
45 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
0
votes
1answer
47 views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
0
votes
0answers
5 views

Manipulating the relative weight of tokens inside CountVectorizer

This is transparently a classic IMDB data recommender question. I'm trying to build a recommender system that suggests movies that a user is likely to enjoy. If I have my terminology correct I am ...
0
votes
0answers
21 views

How could you predict customer churn using transactions date?

I am new to machine learning and I would like to predict churn using dates of transactions. I tried to prepare my data and I couldn't obtain good results. I would like to predict unique Customers, ...
0
votes
0answers
101 views

Encoding with OrdinalEncoder : how to give levels as user input?

I am trying to do ordinal encoding using: from sklearn.preprocessing import OrdinalEncoder I will try to explain my problem with a simple dataset. ...
3
votes
5answers
229 views

How can I handle a column with list data?

I have a dataset which I processed and created six features: ['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList', 'totalProducts'] And the ...
1
vote
1answer
33 views

How do I change the location of Google BigQuery

So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks ...
0
votes
1answer
15 views

Is it acceptable to make new independent variables out of old ones?

I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or ...
1
vote
1answer
21 views

What is the best way to match street addresses in a dataset in Python?

I have a dataset of land parcels owned by the government and I am attempting to match street addresses to an existing list of government agencies. I've used fuzzy matching and used a regex that ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
1answer
24 views

Standardizing features by one specific feature

I am working on a project with a dataset that looks something like the following: ...

1
2 3 4 5
11