New answers tagged data-cleaning
0
For this I would perform two steps, first transforming the data from a wide to long format using pandas.melt (i.e. transform the TP columns over the rows) and then use pandas.pivot to get the desired format.
result = (
df
.melt(id_vars=["POC_Code", "Trading_date"])
.pivot(index=["Trading_date", "variable"], ...
1
If I understand what you need, I think it is this:
b = a.pivot_table(values='TOTAL_BALANCE_EUR', index=['NSFR_GROUP', 'BALANCE_GROUP'], columns='GAP', aggfunc='sum')
b
It's easier for others to help you if you make the data available to others. Just make a tiny dataframe with 10 rows for instance. Also, you can make the code a bit easier to read by ...
0
You can use the pandas.Series.str.contains method to search for all the rows that have the sub-string 'retail' in the 'Segment' column
df.loc[df['Segement'].str.contains('retail', case=False),'Segement'] = 'Retail'
df.loc[df['Segement'].str.contains('corporate', case=False),'Segement'] = 'Retail'
0
Edited my previous comment as there was an Syntax error, This happen as I am new in this join recently(01/04/2021) on this platform
you can try replace function with NumPy library which will help to speed up the process.
df.replace('^^',np.NaN) or
df.replace('not filled in',np.NaN),
df.replace('&&', np.NaN),
df.replace('values needed', np.NaN),
df....
0
You might want to apply one-hot encoding instead. These are not really continuous features. If you consider each day of the week or month of the year a category, then you can instead treat them as categorical variables.
The year is trickier as it does not repeat itself. I would suggest to maybe instead of using the year to use a date difference: which can ...
0
Hello I think these lines could help: my case does not precisely answer to the original question. If we need to keep only the rows having at least one inspected column not null then use this:
from pyspark.sql import functions as F
from operator import or_
from functools import reduce
inspected = df.columns
df = df.where(reduce(or_, (F.col(c).isNotNull() for ...
1
Create a histogram of the watch_time_ms. If you are lucky - you may see a bi-modal distribution (i.e two peaks). The higher/lower peak could be interpreted as interested / uninterested behavior respectively. Then your threshold could lie somewhere in the valley between your two peaks in the histogram.
If your videos are variable length - you may also want to ...
0
I don't want to go through all the new cat and dog photos labelling whether each one is a cat or dog.
This is basically the reason you do ML. Your model predicts category of new cats and dogs from a set of labeled images your model has seen before.
If you mean the distribution of new input might change over time e.g. new pose or style of photos that you did ...
0
If we are looking to remove Non-English words in a column, we can simply do it using regular expressions.
Here is what I tried while cleaning tweets for sentiment analysis-
new_string=re.sub('[^a-zA-Z0-9]',' ',string)
cleaned_string=re.sub('\s+',' ',new_string)
cleaned_string
1
The average or median seem reasonable here. A boxplot might be a good choice of visualistion:
The y-axis could be your time dimension, and boxplot shows the normal distributions of differences, including the median value and outliers. You could then compare certain measurements against this. But it would of course depend on if you have enough data in your ...
Top 50 recent answers are included
