Handling Missing Inconsistent Educational Data

Question

I'm an educational researcher learning about machine learning so I can further explore my data beyond the usual statistics.

I currently have some assessment data but I am not sure how to appropriately handle features with 'no data'. For example, a school offers Subject A, B, C, D, E to students. All students are required to take Subject A and B. Subject C, D, E are optional. This means for Subject C, D, E, instead of a numerical 0-100 grade, it would show only as "-". This is not missing data per se. I can impute it but imputing it with the mean or median does not make much sense because some subjects may only have 5% of the students selecting it.

For Subject A and B, which are compulsory, sometimes students get sick, so a categorical value like 'MC' is recorded instead of a 0-100 numerical grade.

What are the appropriate/best practices to approach data in such scenarios?

Answer 1

Imputing should be done when the data is actually missing, that is, it was supposed to be recorded in data collection process but somehow the value could not be captured. For variables with values like "-" can be filtered along with the set of other variables. In building the machine learning model such variables can be excluded. However, exclusion will depend upon the degree of missingness. Suppose a dataset with 4 variables, 'A, B, X, D'. Suppose, variable 'X' has 80 percent values coded like "-", and then depending on the importance or effect or contribution this variable has for the question to be solved, it can be removed or used. This brings me back to answer the question OP asked, "how to deal with such missingness?". I think a good strategy will be to ascertain what kind/type of value this variable was supposed/meant to contain/hold. Look at the remaining values that are not missing in the variable 'X'. Are they continuous, categorical (ordinal or nominal)? Basis of it, you can then statistically justify the imputation.