# How to handle out of scope value?

I have a data set in which the score column has to be between 0 to 100 and the subject column has to be one of ['Math','Science','English']. However, my data set has different values for some rows.

How should I handle those rows?

  subject score ...
1 Math    90    ...
2 Science 85    ...
3 English 105   ...
4 Comp    95    ...
5 Math    80    ...
6 Science 70    ...

• Simple: you should tell the English teacher that they made a mistake ;) – Erwan Sep 30 '19 at 0:53
• More seriously, it depends what you want to do with this data: it might not matter for your application, in which case you leave them as they are. Or it might matter, in which case you standardize them. You need to give more detail about the context if you want people to help you. – Erwan Sep 30 '19 at 0:56

If you need to clean your data, you can either drop the rows which contain invalid values, or try to correct them.

Here two examples:

# If you want to change the score, so values below 0
# are changed to 0 and values above 100 are changed
# to 100 you can do that like this:

df['score']= df['score'].clip(0, 100)

# Or alternatively (in case you have more complicated
# operations, you can also use where. For the
# correction of the scores, this would look like

df['score']= df['score'].where(df['score']<=100, 100)
df['score']= df['score'].where(df['score']>0, 0)

# If you want to drop the rows that contain undefined
# subjects, you can do that as follows:

valid_subjects= ['Math','Science','English']
# define an indexer that contains True for all rows which are invalid
invalid_subj_indexer= ~df['subject'].isin(valid_subjects)
# now drop them
df.drop(invalid_subj_indexer.index[invalid_subj_indexer], inplace=True)


The result of this looks like:

   subject  score  ...
1     Math     90  ...
2  Science     85  ...
3  English    100  ...
5     Math     80  ...
6  Science     70  ...


You can test the lines above by executing the following lines first to create the test dataframe:

import io
import pandas as pd

raw=\
"""  subject score ...
1 Math    90    ...
2 Science 85    ...
3 English 105   ...
4 Comp    95    ...
5 Math    80    ...
6 Science 70    ..."""