0
$\begingroup$

First post, so thank you in advance for any pointers! I have a dataframe with an issue similar to the following:

id value
10 25.03
11 26.00
11 0.95
12 32.95
13 34.43
13 1.02

One can see that for id's 11 and 13 there are repeats with values that are clear outliers. How would I filter out these outlying values (in the actual dataframe, as here, the outliers are consistently smaller than the "correct" values)? Any tips? Sorry if this needs more clarity, ask and I will try to provide.

$\endgroup$

2 Answers 2

0
$\begingroup$

You can use pandas' drop_duplicates method as such:

df.drop_duplicates(subset=["id"], keep=...)

and by adding proper values for the other argument, specially the keep argument, see the documentation for options.

$\endgroup$
0
$\begingroup$

Assuming you always want to retain the row with the maximum value for each ID, you can use the groupby() method followed by the max() method:

df.groupby(['id']).max()

This will firstly group all the rows with the same id together, then for each group, return the maximum value.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.