To date, I have done several ad-hoc text clustering projects which use combinations of topic modeling, k-means, and other algorithms. Basically, the point of these projects was to produce themes for different events based on associated text. The themes were named manually after the appropriate levels of clustering were determined, and are now stored in a csv in the following format:
event_id majortheme minortheme majortheme_id minortheme_id
12 Job Failure TWS Issue 1 major1minor1
14 Job Failure TWS Issue 1 major1minor1
15 Job Failure Job Abend 1 major1minor2
16 Access Issue Unable to Login 2 major2minor1
17 Access Issue Unable to Connect 2 major2minor2
I want to transition from clustering to classification (from descriptive to prescriptive analytics), i.e. being able to take new events (with new event_id
s) and classify them based on previous clusterings. This would be sort of an iterative training dataset, as new classifications would be added to previous clusterings, after verifying that the model hasn't gone completely awry. Using Python, what is the best approach to implement this sort of classification pipeline? Is it as simple as saving my initial clustering results and then just using that data as a training set going forward? Then saving the test predictions to the original training dataset, and so on?