Let's say I have been given 1000 documents and 6 labels from someone. My job is to label each of these 1000 documents into 1 of the 6 labels which are words not numbers. How can I automate or semi-automate this process using data science??
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
Semi-supervised learning. You label 1% manually, let the algorithm learn, then it labels unknown data, learns from it and labels again.
$\begingroup$
$\endgroup$
You have two options. Supervised learning where you will have to label the data manually and then use those data points to train a model and predict the remaining instances.
Or, you can use unsupervised learning, these are techniques which do not need a label. You can use k-means to cluster your data into $k=6$ labels. Then you can associate these clusters with the label based on your experience.
-
$\begingroup$ How to use k-means...the centroids are initialised randomly so they won't cluster the documents according to my labels? Will it be right here to not initialize centroids randomly?? $\endgroup$ – Rishabh Baid Nov 1 '18 at 6:26
-
$\begingroup$ It's best to randomly initialize them to avoid introducing bias. Let the centroids converge. Then attribute each cluster with one of your labels. $\endgroup$ – JahKnows Nov 1 '18 at 6:29
-
$\begingroup$ I tried doing exactly that..But I find at the end of the K-means program(random initialization) that there are a number of documents in the same cluster which wouldn't be in the same cluster...if I was doing the labeling manually. $\endgroup$ – Rishabh Baid Nov 1 '18 at 8:12
