4 votes

Converting YYYYQQ to YYYY-MM-DD

Since your end goal is just datetime64, use replace with to_datetime. Given a Series of <...
  • 229
4 votes

Imputing time data for an event that hasn't occurred yet

As so often, the answer is: "It depends". In this case, it depends on the algorithm / model / method you are going to use. There are some methods, e.g. tree-based methods, that can handle ...
  • 91
3 votes

Imputing time data for an event that hasn't occurred yet

It's very easy to over-complicate things. Agree with Broele that it depends. Let's look at what data you actually have: Since t0 (when you started collecting data), you know this location has not had ...
  • 413
2 votes

How to remove outliers properly?

Yes, the problem of imbalance is indeed genuine while pre processing. There are no hard and fast rules for removing outliers, but generic methodologies (percentile,boxplot,Z-score etc). Like gender, ...
  • 21
2 votes
Accepted

test data is not a good representation of train data

Here's is my attempt to answer these questions: Avoid taking any insights from test_data. Do changes WRT the insights taken from the train_set only. However, every change I make should be replicated ...
  • 357
2 votes

Deleting a Column in a csv or Excel file using Pandas

The code master_df.drop(["Film Number"], axis=1, inplace=True) you have written is right. What is happening is like you have removed the column perfectly ...
2 votes
Accepted

Joining on columns with duplicate values - clean before merging or after merging?

With small datasets it doesn't matter, but for large datasets it is always better to remove duplicates before joining, just for efficiency. There is usually an increase in CPU time when you are ...
2 votes
Accepted

Drop or impute the missing values?

In most cases, dropping data only makes sense when you have a large number of nan values. For example of you have a feature with 98% nan values, it is not going to be of much use to any algorithm. ...
  • 1,293
2 votes

Category under which smote falls into

SMOTE should not appear on there, as it is a method for synthesizing new points, not for making predictions based on existing features. Also, the need for SMOTE is contested. Finally, I dispute the ...
  • 3,278
2 votes

Feature Engineering on transactional dataset clustering

Depending on your processing limitations I'd be tempted to do the following, for each client a single row consisting of: Age Sex (one hot encoded) Per store: Purchase Count Total Purchase Value ...
1 vote
Accepted

Are you supposed to clean new data before it is fed to a machine learning model?

Yes, it makes perfect sense to clean/preprocess the new data much like train /test dataset. For reference: [https://stackoverflow.com/questions/66301306/do-you-have-to-clean-your-test-data-before-...
  • 319
1 vote
Accepted

Denoising in ML Pipeline

Not sure what kind of data you have. In case your using images, is done by image, in which case, you wouldn't have data leakage if you split before or after. But if your talking about a time-related ...
1 vote

Converting voting district GeoID to approximate zip code?

The first 5 digits of the GEOID are the FIPS code, and you have the full list available here linked with the ZIP code, county name, etc. https://www.kaggle.com/datasets/danofer/zipcodes-county-fips-...
1 vote

Converting voting district GeoID to approximate zip code?

Voting districts can be matched to ZIP codes by their coordinates. To get an approximate result I suggest the following algorithm: Calculate voting district coordinates by averaging its polygon ...
1 vote
Accepted

How to convert duration column from 1 hr. 17 min to 77 min in pandas?

You are replacing the spaces with a + character before replacing `` and . Therefore the minutes and seconds do not get replaced, which in turn cannot be evaluated ...
  • 6,169
1 vote

Joining on columns with duplicate values - clean before merging or after merging?

This is similar to the discussion here. So, it depends. You just need to aim for readability. I would argue for the cleaning-first approach however, so that the merging operation is done on smaller ...
1 vote

Performing EDA on a dataset with missing features

There are a lot of techniques through which you can fill the missing values. Some of them are: 1.) Replacing with mean, median or mode as you correctly pointed out. 2.) Replacing with a constant value ...
  • 1,293
1 vote
Accepted

Data Wrangling and data cleaning

Data Wrangling Data wrangling also referred to as data munging, is the process of converting and mapping data from one raw format into another. The purpose of this is to prepare the data in a way that ...
  • 2,466
1 vote
Accepted

Can I create a new target value based on the average target value of same data points for regression?

Your solution makes total sense and if you do not have temporal data in production then this is how you better do. I just add small points: Data Leakage does not happen when you transform solely ...
1 vote

Merging 2 datasets

Test data should never be used in the training process. The reason why there are test data ist that you test your model performance against data the model has never seen before. By doing so, you "...
  • 6,962
1 vote

Why is an ML algorithm performing better with correlated features, than the one with uncorrelated ones?

Few points to not here: Multicollinearity effects linear model much more as compared to Random forest as it is picking up different set of features (read sampling with replacement) for every model ...
  • 1,713
1 vote
Accepted

Improving text classification & labeling in imbalanced dataset

why categories are converted to numeric values? Its due to the simple fact that the most machine learning models do not accept categorical values to perform prediction. For this reason its Yes, for ...
1 vote

If I have two variables with strong correlation, should I delete one and leave the other in my data

It depends. If you are using this data on a linear model it is better to remove correlated features. But some non-linear complex model can use or eliminate these correlated feature automatcially.
  • 743
1 vote

how to deal with columns that has different value in only 1 or 2 rows?

In other words, you have sparse binary features. A vast majority of the data is zeros. The remaining data are ones. One option is to transform the features to be denser. This can be done with ...
1 vote
Accepted

Tidy vs. untidy data

As you mention, the first example is data in an "untidy" format which can make analysis more difficult because of multiple reason. The first one being the column names, as you mention you do ...
  • 6,169
1 vote

Logic Behind IQR Outlier Detection

IQR says observations in a feature that are "too far" from the median are outliers. There are many many algorithms to find "outliers". Here are a few - here and here. However ...
  • 864
1 vote
Accepted

Data snooping and information leakage?

While technically this may be a situation of information leakage since you're applying a pre-processing step on the whole dataset before performing a dataset split, I don't think it matters too much ...
  • 6,169
1 vote

Data Preparation for dates

Given that you cannot modify the data, I assume you are looking for data analysis techniques that can highlight a systematic pattern? If so we need more info. Make use of context of the problem. Is ...
  • 331
1 vote

How to remove rows from a data frame that have special character (any character except alphabet and numbers)

You can use the regular expression to clean your data. Below I have compiled an almost complete list of functions that one uses frequently when cleaning text data. ...
  • 1,293
1 vote
Accepted

How to find median/average values between data frames with slightly different columns?

Assuming that you have the data stored in separate dataframes, you can use a combination of pandas.concat and ...
  • 6,169

Only top scored, non community-wiki answers of a minimum length are eligible