# How to plot High Dimensional supervised K-means on a 2D plot chart

I'm Having a ML problem where my data set contains 80 features labelled into 3 groups (0, 1, -1).

I want to plot the data on a 2D surface to see how "close" (similar) data with label x is to data with label y, how the data spreads, are the labels separable, etc.

I was thinking about using PCA and transform the data from 80D to 2D, but It only retain 40% of the variance!

• Is this a good approach for the problem?
• If so, does 40% suffice?
• Are there any other/better approach for this?

## EDIT:

Plotting is not the main issue. The transformation from 80D to 2D (for an easy visialization) is whats difficult.

Also, all of this is being made to know how much samples with label 1 differs from label 0 and label -1 and vice versa (based on those original 80 features).

If there's a different method, that is not visualizing the "answer", I'll also be happy to hear about it!

What you are looking to do is perform some projection or feature compression (both of those terms mean the same thing in this context) onto a 2D plane while maintaining relative similarity. Many of these techniques exist each optimizing a different aspect of relative "closeness".

The following code will show you 4 different algorithms which exist which can be used to plot high dimensional data in 2D. Although these algorithms are quite powerful you must remember that through any sort of projection a loss of information will result. Thus you will likely have to tune the parameters of these algorithms in order to best suit it for your data. In essence a good projection maintains relative distances between the in-groups and the out-groups.

The Boston dataset has 13 features and a continuous label $Y$ representing a housing price. We have 339 instances.

from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.manifold import TSNE, SpectralEmbedding, Isomap, MDS

X = boston.data
Y = boston.target

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

# Embed the features into 2 features using TSNE# Embed
X_embedded_iso  = Isomap(n_components=2).fit_transform(X)
X_embedded_mds  = MDS(n_components=2, max_iter=100, n_init=1).fit_transform(X)
X_embedded_tsne = TSNE(n_components=2).fit_transform(X)
X_embedded_spec = SpectralEmbedding(n_components=2).fit_transform(X)

print('Description of the dataset: \n')

print('Input shape : ', X_train.shape)
print('Target shape: ', y_train.shape)

plt.plot(Y)
plt.title('Distribution of the prices of the homes in the Boston area')
plt.xlabel('Instance')
plt.ylabel('Price')
plt.show()

print('Embed the features into 2 features using Spectral Embedding: ', X_embedded_spec.shape)
print('Embed the features into 2 features using TSNE: ', X_embedded_tsne.shape)

fig = plt.figure(figsize=(12,5),facecolor='w')
plt.subplot(1, 2, 1)
plt.scatter(X_embedded_iso[:,0], X_embedded_iso[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using Isomap \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.tight_layout()

plt.subplot(1, 2, 2)
plt.scatter(X_embedded_mds[:,0], X_embedded_mds[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using MDS \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.show()
plt.tight_layout()

fig = plt.figure(figsize=(12,5),facecolor='w')
plt.subplot(1, 2, 1)
plt.scatter(X_embedded_spec[:,0], X_embedded_spec[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using Spectral Embedding \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.tight_layout()

plt.subplot(1, 2, 2)
plt.scatter(X_embedded_tsne[:,0], X_embedded_tsne[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using TSNE \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.show()
plt.tight_layout()


The target $Y$ looks like:

The projected data using the 4 techniques is shown below. The color of the points represents the housing price.

You can see that these 4 algorithms resulted in vastly different plots, but they all seemed to maintain the similarity between the targets. There are more options than these 4 algorithms of course. Another useful term for these techniques is called manifolds, embeddings, etc.

Check out the sklearn page: http://scikit-learn.org/stable/modules/classes.html#module-sklearn.manifold.

You can take a look at this answer in Cross Validated : https://stats.stackexchange.com/a/53194/211338

It explains common techniques to visualize and compare high dimensional data. As you have 3 classes, just plot the different plots 3 times or compare them classes in "0 vs 1", "0 vs -1" and "-1 vs 1" plots, depending on the case.

EDIT

I don't really see why you would like to perform feature reduction, nor when. You wrote you want to use K-means on that data (60D data) and then just plot it on 2D plots to visualize the similarity of the labels. Well then, do it...

You can apply K-means on your 60D data (supposing it has already been cleansed and properly analyzed), so you don't loose information. That will yield the class for each observation (0, -1 and 1) and voilà.

If you really want to perform feature reduction: If you have some knowledge of the data, try to use other technique than PCA (search for Exploratory Data Analysis). If your data is anonymous, you can then perform PCA or any other type of dimensionality reduction method, like t-SNE (which should perform better on lots of cases).

As you have noticed, PCA does not always yield amazing results. 60% of information seems not enough (to me). This means 40% of the information has been lost and will not be used in your algorithm. You can try and take some more principal components. I use an 85% treshold when I have a lot of instances and 95% when the dataset is rather small.

• Mate, I'm sorry I misinformed you. I've dropped the k-means part. Commented Aug 22, 2018 at 11:33