1
$\begingroup$

I have a set of 1000 models. Each model is a (72 x 4) matrix, where 72 are the values associated to each of the 4 variables. The goal is to perform cluster analysis on these models, i.e. to group them based on a similar distribution of all the 4 variables. It turns out that this is a multidimensional (72) and multivariate (4) problem.

I am trying to implement it in Python by using the k-means algorithm implemented in tensorflow module. First, I have transformed the input dataset to a ndarray (1000 x 288), where the rows are the models and the columns the 288 = 72 x 4 values (this should be fine, since I need to perform cluster analysis based on all the variables). In a sense, I have addressed the multivariate issue by converting the problem to a "univariate" problem.

As a second step, I have standardized the array, so to map the variable values to a common scale.

models = np.array(models)
from sklearn.preprocessing import StandardScaler
   
scaler = StandardScaler()

models_scaled = scaler.fit_transform(models.transpose())
models_scaled = models_scaled.transpose()

import tensorflow as tf

#def input_fn():
    #return tf.constant(models_par, dtype=tf.float32)

def input_fn():
  return tf.compat.v1.train.limit_epochs(
      tf.convert_to_tensor(models_scaled, dtype=tf.float32), num_epochs=1)    

num_clusters = 5
kmeans = tf.compat.v1.estimator.experimental.KMeans(num_clusters=num_clusters, relative_tolerance=0.001)    

# train
num_iterations = 20
previous_centers = None
for _ in range(num_iterations):
  kmeans.train(input_fn)
  cluster_centers = kmeans.cluster_centers()
  if previous_centers is not None:
    print('delta: ', (cluster_centers - previous_centers))
  previous_centers = cluster_centers
  print('score: ' , (kmeans.score(input_fn)))
print('cluster centers: ' , (cluster_centers))

# map the input points to their clusters
cluster_indices = list(kmeans.predict_cluster_index(input_fn))
for i, point in enumerate(models_scaled):
  cluster_index = cluster_indices[i]
  center = cluster_centers[cluster_index]
  print('point: ' , (point) , 'is in cluster ' , (cluster_index) , 'centered at ' , (center))

Assuming this is all correct up to this point, I have troubles in visualizing the output returned by tensorflow. I am trying to get a parallel coordinate plot:

X_clustered = pd.DataFrame()
X_clustered['feature_vector'] = models_scaled.tolist()
X_clustered['cluster'] = cluster_indices

The code above is obviously wrong. Although I have solved (?) the multivariate aspect of the problem, the issue is it's still multidimensional. You may argue I could reduce dimensionality by implementing PCA, for instance, but for the parallel coordinate plot I think the original dataset is required.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.