How to use a dataset where attribute names are changed?

Question

I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed.

My goal is to use dataset to decide whether new credit requesters request is approved or not, base on the data provided by them.

I see this dataset is widely use, but I do not know use It without knowing the attribute names?(I am aware that some attribute values can be changed to protect confidentiality)

What piece of knowledge am I missing here to use this dataset?

Answer 1

What piece of knowledge am I missing here to use this dataset?

By anonymising the data and altering the attribute names, the providers of this dataset have made it into an abstract machine learning exercise, where data has similar qualities to real-world credit approval data, so that approaches to train models that work well on it are likely to work well in a real-world scenario. However, those models could not be used in a production system. You are not in a position to train a model based on it and then input real world data you have collected elsewhere to make a prediction.

You can still train a model on this data to predict A16 (the class attribute). You can measure the accuracy - or any other metric - by holding out a test set. You can experiment with feature engineering, feature selection etc, in an abstract way without being able to apply much domain knowledge. You can try out different model classes, different hyper-parameters, different approaches to imputing missing values or cross-validation etc.

Your project can draw conclusions about the approach you have taken, based on test results. What you cannot do is explore real-world scenarios with simulated customer data. This might make it less compelling as a demonstration - for instance users could not explore exactly what factors might lead to an application being approved or not, instead you would just have to show some graphs that demonstrate you have selected a model with good performance.

Answer 2

I'll start with what you want to know, and move to the caveats. You are probably safe to use any number of decision tree algorithms to do some black box analysis, given the peer-reviewed publications on the dataset using them (though as any academic will tell you, this is often a stretch too). As the papers mention, C4.5, CART, Random Forrest, adaboost, and the like are all candidate models. Most can be used to gather a variable importance ranking for predicting the response variable you chose, and you should be able to examine confusion matrix metrics for evaluating performance if you chose to predict class variables. Try different pruning methods on the tree(s) to prevent overfit, and maybe even nest models to build an ensemble. Don't forget to rigorously validate the model (I recommend cross-validate, which is baked into most frameworks that handle decision trees).

However, I urge you against using this data. Here are a few reasons why:

1. Diagnostics nightmare. Just about all modeling techniques rely on certain assumptions. Some common ones that can break most models are the presence of autocorrelation, heteroskedasticity, and multicollinearity. Violate these, and most models will become unstable. You can try controlling for this by plotting ACFs and PACFs, throwing out highly correlated variables or lumping them together some way, just to name a few approaches. But these all have problems. The problem with throwing out highly correlated variables is that you might throw out the most important relationships that in fact don't violate anything! Or, say your data turns out to be a multivariate time series? The PACF no longer interprets the same way. Trees, used with this data set, are fairly robust, but not perfect. Their use in literature likely means that you can ignore this, but be cautious, as some papers explicitly use data that don't work to make a point to about how algorithms can break. Remember the cardinal rule: garbage in, garbage out. Best advice here is to read the papers (though this is generally not something you can fall back on with other data).
2. Causality is impossible to detect. Blind data makes it near-impossible to track things temporally, which means that the model is likely to end up using variables in it's predictions that are only known after the event. Conceptually, this isn't very useful unless your model will only ever be used to fill missing bits in already collected data.
3. Interpretation. This might pass for a school project, but in most cases interpretation is everything. Ask this question: what can the client (in this case, the credit company) do with your analysis? There is no way to know with masked data. Say you built a black box model with a decisin tree. It does a fantastic job of predicting the outcome. So, you put a bow on it and send it to the client. But what happens when they unwrap it, renaming the variables? It can potentially be useless, and you have no way of knowing. You may be predicting an unimportant variable, or drawing attention to "insights" that are completely obvious. An analyst's worth is in what he brings to the table, not the models he uses.

In summary, read the papers, use trees, and fear unknown data. There be dragons.