Neural Networks: How to prepare real world data to detect low probability events?

Question

I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records are incomplete or contain outliers. The dependent variable is Defaulted / Good (0,1). We are trying to train a neural network to predict defaults based on the training data. I have experience with neural networks and used them for sample data with great results, however, I never had to normalize noisy real world data.

Any thoughts what I should keep in mind in respect to: - How to normalize the categories. Can I assign an indexed number? Should I tried to stratify them? - How to deal with missing data. Assign 0 ? - How to deal with the fact that defaults are only about 5 % of the data set. What transfer function would be useful to predict these low probabilities. - Basically any other real world data advice is very much appreciated.

Thanks in advance!

Answer 1

You bring up a number of good questions here Ans. I will do my best to cover each of them in turn. It isn't an exhaustive treatment but hopefully it helps...

1. How to normalize the categories.

First, assess whether your categorical variables can be considered zero variance (e.g. all records possessing one category only) or near zero variance (vast majority of records belonging to very few categories). Create a basic frequency distribution to identify this.

While it doesn't matter as much in Neural contexts per se, it is a good idea to consider filtering low variance variables from your model. Just be careful as eliminating near zero variance variables may have you throwing out the baby with the bath water.

You've worked with Neural Nets before so you know that you need to convert categories to numeric values. A good question to ask is whether a given categorical value is ordinal in nature (e.g. on a Likert scale of 1 to 5) and whether you want to preserve ordinality. This pushes you into an area such as that described by Pinto da Costa and Cardoso (https://www.researchgate.net/publication/221112186_Classification_of_Ordinal_Data_Using_Neural_Networks).

2. How to deal with missing data.

Assuming that you are talking about missing continuous values, you will want to impute these numeric values based collectively on the values that are present across the entire attribute. There are a number of approaches to use here but something important to keep in mind is dispersion. In a basic sense, if your attribute is skewed by outliers, you will want to steer clear of a mean-based calculation and go with a median-based approach.

3. How to deal with the fact that 'Defaults' are ~5% of the data.

This isn't an immediate concern given that you have 2500 examples of what denotes a 'Default' (in a 50,000 dataset). A sane approach here would be to ensure that you are using a K-fold cross validation scheme (say with 10 folds) to ensure that you are truly randomizing training vs test. This will help protect you against overfitting. Again, this is pretty high level guidance, but it is prudent.

Extra Credit: I haven't gotten into other standard tactics such as normalization of your continuous attributes, but you may want to get up to speed there to better generalize your model (while also developing a deeper understanding of the dynamics at play in the data). This could point you to trying out other algos, etc.