I would like to remove outliers from my dataset. It looks like this:
time Gbps
0 2018-11-20 00:00:00 29.821748
1 2018-11-20 01:00:00 38.620987
2 2018-11-20 02:00:00 0.010000
3 2018-11-20 00:00:00 29.821748
4 2018-11-20 01:00:00 38.620987
5 2018-11-20 02:00:00 0.010000
As you take a look at this table, you can see that number 5 and 2 are the outliers. I wrote a interquartile range (IQR) method to remove them. However, it does not work. I don't know if I do something wrong in Pandas/Python, or it's the fact I do something wrong in statistics. Any ideas? The result from this function is the same frame as I had at the beginning.
def IQR(data):
q1 = data['Gbps'].quantile(0.25)
q3 = data['Gbps'].quantile(0.75)
iqr = q3 - q1
fence_low = q1 - 1.5 * iqr
fence_high = q3 + 1.5 * iqr
cleaned_data = data.loc[(data['Gbps'] > fence_low) & (data['Gbps'] < fence_high)]
return cleaned_data
data = {
'time': ['2018-11-20 00:00:00', '2018-11-20 01:00:00', '2018-11-20 02:00:00', '2018-11-20 00:00:00', '2018-11-20 01:00:00', '2018-11-20 02:00:00'],
'Gbps': [29.8217476333333333, 38.6209872666666667, 0.01, 29.8217476333333333, 38.6209872666666667, 0.01]
}
df1 = pd.DataFrame(data, columns = ['time', 'Gbps']
cleaned1 = IQR(df1)
print(cleaned1)