Cluster analysis as an associative model?

Question

I have a set of data with many samples and many features, but where half of the data is missing one variable (call it A), which is composed of four categories. Based on the half of data which has A, I want to know what category the samples without A would most likely be in if they did have A.

I could build a classifier based on the data with A, and predict the data without A (this is the best route IMHO).

But I'm wondering, out of curiosity, if this method could also be a very, very, very rough way of doing something similar:

1. Cluster the data which has A into the same number of clusters as categories in A (in this case four).

2. Check for an an association between the clusters and the categories in A (using a frequency table and chi-square test).

3. If there is an association, run the data without A through the clustering model to figure out what category of A it's most likely associated with (based on what cluster it is in).

Thoughts?

Answer 1

I think the question you're wrestling with is essentially this: Is there a way to use information that may be present in the data without A as part of your strategy for predicting A? There is actually a name for the set of methods that do exactly that: semi-supervised learning.

While there are multiple techniques, a method analogous to what you suggest in your question would be something like the following. Here I refer to the set of data with A as labeled and the rest as unlabeled.

1. Apply an unsupervised learning technique (such as clustering) to the full data set (both labelled and unlabelled).

2. Transform the labelled data based on the result of your unsupervised learning technique.

3. Apply a supervised learning technique with the transformed set (possibly combined with the original set) of labelled data.

Generally you would want to apply a cross-validation strategy as well to detect overfitting.

Answer 2

I try to stay away from bounty problems as they seem to lead to one upmanship, but I can't resist this one as its an interesting question...

The upper classification method that your describe sounds spot on. Use that one!

The bottom method has a number of issues:

How would you know that the clusters you get out would correspond to the categories of A? The answer is that you wouldn't. There are other data features and the clusters would be a linear superposition of the maximum separation between the various variables.

I guess you could standardize your data and then coax the clusters to conform to the separation along A by multiplying the A feature by a factor greater than 1 (~2-10), but this seems like more of a bastardization of clustering than a good process.

The main issue that I see with your method is that you are essentially employing a semi-supervised learning methodology, but you don't need to because you have the data to solve this as a supervised learning problem. It would be a good methodology if you only had a handful of labeled A values, but since you have lots more data, you should harness the power of that data that you have. The supervised learning problem will fully harness the variance of your data and much of that will be thrown out in the semi-supervised learning methodology.

Hope this helps and good luck!

Answer 3

Clustering techniques at high dimensionality tend to be unstable/inaccurate. As such, if you are able to accurately classify one column based on others, then you are well suited with pursuing a dimensionality reduction technique such as Non-negative matrix factorization. In this way, you can simply remove the column from all data (as the other columns hold the same information), and then cluster on the reduced space.