I have a semi structured data set. I need to collect some data (unlabeled) randomly for labeling.

As initiative at first I separated labeled and unlabeled data. Then I convert those data from string to numeric as all data were string.

After that I cluster unlabeled data into 2(cluster 0 and cluster 1) by applying k-means clustering algorithm.

Now when I collect some data randomly from both cluster it takes indexes from both labeled and unlabeled data. But my expectation was, indexes will come just from unlabeled data as cluster was created by using unlabeled data.

I couldn't figured out the problem. I am giving my code below.


enter image description here

The indexes of labeled data stats from 0 to 343 (344 data) and unlabeled data starts from 344 to 863 (519 data)


enter image description here

# define X and y
feature_cols = ['TopLeft', 'TopMiddle', 'TopRight', 'MiddleLeft','MiddleMiddle', 'MiddleRight', 'BottomLeft', 'BottomMiddle', 'BottomRight']                   

# X is a matrix, hence we use [] to access the features we want in feature_cols
X =train_unlabeled[feature_cols]`

# y is a vector, hence we use dot to access 'label'
y =train_unlabeled['Class']

X_scale = scale(X)
reduced_data = PCA(n_components=None).fit_transform(X_scale)
kmeans = KMeans(init='k-means++', n_clusters=2, n_init=10)

enter image description here


Now I was trying to collect 95% data from each cluster (cluster was created by train_unlabeld data)

x = 0.95
C_i = np.where(labels == i)[0].tolist() 
n_i = len(C_i)
sample_i = np.random.choice(C_i, int(x * n_i)) 
print (i, sample_i)
x = 0.95
C_i = np.where(labels == i)[0].tolist() 
n_i = len(C_i)
sample_i = np.random.choice(C_i, int(x * n_i)) 
print (i, sample_i)

When I print the index while collecting data randomly depending on cluster(0 and 1) I found these indexes

enter image description here

As the index of unlabeled data is started from 344, so how the index of labeled data comes in cluster 0 and ? I didn't understand it.

  • $\begingroup$ Can you please rephrase your question, it's very hard to understand what you are trying to get at here. $\endgroup$ – JahKnows Apr 24 '18 at 8:11
  • $\begingroup$ You seem to have labelled data. In that case you should train a model using that data and then apply your model to classify the unlabeled instances. Can you explain why you are attempting to cluster unlabeled instances and then applying that to labeled data? $\endgroup$ – JahKnows Apr 24 '18 at 8:13
  • $\begingroup$ Yeah and thanks @JahKnows for your answer. I know that I can labeled those unlabeled data by classification algorithm but I don't need to labeled all unlabeled data I just need some unlabeled data for labeling that's why I collect some data randomly. $\endgroup$ – IS2057 Apr 24 '18 at 8:20
  • $\begingroup$ What are you trying to do in general with this data? You want more instances in order to train a better model? $\endgroup$ – JahKnows Apr 24 '18 at 8:23
  • $\begingroup$ Sorry, JahKnows, I couldn't disclose everything what my purposes is. But I need some unlabeled data that's why at first I clustered them and was trying to pick some data from that cluster. $\endgroup$ – IS2057 Apr 24 '18 at 8:32

The problem was understanding the index between train_unlabeled data and the cluster. There are no connection between index of train_unlabeled data set and the data that belongs to cluster. As I am using jupyter notebook, so the indexes given to itself. Now when I clustered those unlabelled data (index start from 344 to 863) into 2, then which data belongs which cluster it already takes its new index for cluster which is not same as previous index.

For better understanding I'm giving an example:

suppose for unlabelled data (train_unlabeled) the data those indexes are 344,345,346,347 and 348. After clustering 344 and 346 took their place at cluster 0 and 345,347,348 at cluster 1. While clustering the new index of 344 will be 0, for 345 will be 1, for 346 will be 2 and so on.

That's why when I print the indexes of data, belongs to cluster 0 and 1 (by using print (i, sample_i) this command) it just shows the index of data of its own cluster.

My mistake was I though that both indexes are same.

Now for keep similarity of indexes before and after cluster, I reset the the index of train_unlabeled data.

The command is: train_unlabeled = train_unlabeled.reset_index()

Here comes another problem, that is a new column named:index is added.

enter image description here

So, after resetting index (now, index starts from 0), I drop the extra index column by using this command : train_unlabeled =train_unlabeled.drop('index', axis=1, inplace=True)

And at last I cluster that unlabeled data (train_unlabeled) and would be able to take 95% data randomly from cluster by using cluster index.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.