I have a set of 1000 models. Each model is a (72 x 4) matrix, where 72 are the values associated to each of the 4 variables. The goal is to perform cluster analysis on these models, i.e. to group them based on a similar distribution of all the 4 variables. It turns out that this is a multidimensional (72) and multivariate (4) problem.
I am trying to implement it in Python by using the k-means algorithm implemented in tensorflow module. First, I have transformed the input dataset to a ndarray (1000 x 288), where the rows are the models and the columns the 288 = 72 x 4 values (this should be fine, since I need to perform cluster analysis based on all the variables). In a sense, I have addressed the multivariate issue by converting the problem to a "univariate" problem.
As a second step, I have standardized the array, so to map the variable values to a common scale.
models = np.array(models) from sklearn.preprocessing import StandardScaler scaler = StandardScaler() models_scaled = scaler.fit_transform(models.transpose()) models_scaled = models_scaled.transpose() import tensorflow as tf #def input_fn(): #return tf.constant(models_par, dtype=tf.float32) def input_fn(): return tf.compat.v1.train.limit_epochs( tf.convert_to_tensor(models_scaled, dtype=tf.float32), num_epochs=1) num_clusters = 5 kmeans = tf.compat.v1.estimator.experimental.KMeans(num_clusters=num_clusters, relative_tolerance=0.001) # train num_iterations = 20 previous_centers = None for _ in range(num_iterations): kmeans.train(input_fn) cluster_centers = kmeans.cluster_centers() if previous_centers is not None: print('delta: ', (cluster_centers - previous_centers)) previous_centers = cluster_centers print('score: ' , (kmeans.score(input_fn))) print('cluster centers: ' , (cluster_centers)) # map the input points to their clusters cluster_indices = list(kmeans.predict_cluster_index(input_fn)) for i, point in enumerate(models_scaled): cluster_index = cluster_indices[i] center = cluster_centers[cluster_index] print('point: ' , (point) , 'is in cluster ' , (cluster_index) , 'centered at ' , (center))
Assuming this is all correct up to this point, I have troubles in visualizing the output returned by tensorflow. I am trying to get a parallel coordinate plot:
X_clustered = pd.DataFrame() X_clustered['feature_vector'] = models_scaled.tolist() X_clustered['cluster'] = cluster_indices
The code above is obviously wrong. Although I have solved (?) the multivariate aspect of the problem, the issue is it's still multidimensional. You may argue I could reduce dimensionality by implementing PCA, for instance, but for the parallel coordinate plot I think the original dataset is required.