I'm a complete newbie to the world of machine learning, and I'm currently working on implementing a model that will need to incorporate feedback, as well as changes to the data set (both feature & label changes over time).
The frequency of change isn't yet entirely known, but for simplicity could probably be rolled into a batch every day or so.
I'm aware of how I can build a training & test set, and get a classifier up and running.
My primary issue is that it's probably not going to be ideal to run a completely fresh training every time there's a change.
Users will be interacting with the system via "this was helpful / not helpful" type feedback, which I want to use to strengthen / weaken its association model.
I'm absolutely in the dark as to how once you have the model from the initial data, you can then get it to refine over time from this sort of feedback, and how to update (i.e. add/remove features & labels) without starting from scratch.
tl;dr:
What sort of classifier is best suited to this sort of refinement-over-time problem?
I'll also add that the model needs to support multi-label classification, so any caveats / gotchas / information on how to do this in the broader context of my question would be helpful too.