Let's say I have been given 1000 documents and 6 labels from someone. My job is to label each of these 1000 documents into 1 of the 6 labels which are words not numbers. How can I automate or semi-automate this process using data science??
Semi-supervised learning. You label 1% manually, let the algorithm learn, then it labels unknown data, learns from it and labels again.
You have two options. Supervised learning where you will have to label the data manually and then use those data points to train a model and predict the remaining instances.
Or, you can use unsupervised learning, these are techniques which do not need a label. You can use k-means to cluster your data into $k=6$ labels. Then you can associate these clusters with the label based on your experience.
-
$\begingroup$ How to use k-means...the centroids are initialised randomly so they won't cluster the documents according to my labels? Will it be right here to not initialize centroids randomly?? $\endgroup$ – Rishabh Baid Nov 1 '18 at 6:26
-
$\begingroup$ It's best to randomly initialize them to avoid introducing bias. Let the centroids converge. Then attribute each cluster with one of your labels. $\endgroup$ – JahKnows Nov 1 '18 at 6:29
-
$\begingroup$ I tried doing exactly that..But I find at the end of the K-means program(random initialization) that there are a number of documents in the same cluster which wouldn't be in the same cluster...if I was doing the labeling manually. $\endgroup$ – Rishabh Baid Nov 1 '18 at 8:12