I have a set of 1000 models. Each model is a (72 x 4) matrix, where 72 are the values associated to each of the 4 variables. The goal is to perform cluster analysis on these models, i.e. to group them based on a similar distribution of all the 4 variables. It turns out that this is a multidimensional (72) and multivariate (4) problem.

I am trying to implement it in Python by using the k-means algorithm implemented in tensorflow module. First, I have transformed the input dataset to a ndarray (1000 x 288), where the rows are the models and the columns the 288 = 72 x 4 values (this should be fine, since I need to perform cluster analysis based on all the variables). In a sense, I have addressed the multivariate issue by converting the problem to a "univariate" problem.

As a second step, I have standardized the array, so to map the variable values to a common scale.

models = np.array(models)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

models_scaled = scaler.fit_transform(models.transpose())
models_scaled = models_scaled.transpose()

import tensorflow as tf

#def input_fn():
    #return tf.constant(models_par, dtype=tf.float32)

def input_fn():
  return tf.compat.v1.train.limit_epochs(
      tf.convert_to_tensor(models_scaled, dtype=tf.float32), num_epochs=1)    

num_clusters = 5
kmeans = tf.compat.v1.estimator.experimental.KMeans(num_clusters=num_clusters, relative_tolerance=0.001)    

# train
num_iterations = 20
previous_centers = None
for _ in range(num_iterations):
  cluster_centers = kmeans.cluster_centers()
  if previous_centers is not None:
    print('delta: ', (cluster_centers - previous_centers))
  previous_centers = cluster_centers
  print('score: ' , (kmeans.score(input_fn)))
print('cluster centers: ' , (cluster_centers))

# map the input points to their clusters
cluster_indices = list(kmeans.predict_cluster_index(input_fn))
for i, point in enumerate(models_scaled):
  cluster_index = cluster_indices[i]
  center = cluster_centers[cluster_index]
  print('point: ' , (point) , 'is in cluster ' , (cluster_index) , 'centered at ' , (center))

Assuming this is all correct up to this point, I have troubles in visualizing the output returned by tensorflow. I am trying to get a parallel coordinate plot:

X_clustered = pd.DataFrame()
X_clustered['feature_vector'] = models_scaled.tolist()
X_clustered['cluster'] = cluster_indices

The code above is obviously wrong. Although I have solved (?) the multivariate aspect of the problem, the issue is it's still multidimensional. You may argue I could reduce dimensionality by implementing PCA, for instance, but for the parallel coordinate plot I think the original dataset is required.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.