I have no idea whether this is the right StackExchange flavor to post this question in, but here goes:
I have an agent (an IRC bot that listens to an event stream) that I would like to add "intelligent notifications" to. Let me give a specific example.
The bot monitors the results from our continuous integration system, so it basically sees a stream of test results associated with changes (and the changes are associated with users.) If I ask it to notify me of interesting events, then for each event it sees, it decides whether to tell me about it (or rather, about the current results so far.)
I would like it to decide based on how interesting something is for me. Something is interesting if (1) it is a result of a change that I pushed to the continuous integration system, (2) it is a test failure, and (3) the information conveyed by that failure is a significant indicator of whether my change was bad.
Tests have a baseline probability of failing for no good reason ("intermittent failures"). If one test fails, it does not necessarily mean my change was bad. It might just be a flaky test. (Any given push tends to result in a couple bogus failures, so this isn't some obscure edge case.) But the baseline probability of an intermittent failure varies according to the test suite running (we have several dozen different test suites that fire off for every change pushed.)
So I don't want to be bothered with reports of failing tests that are probably just noise. And if my push is bad, I don't want to be flooded with notifications for every failed test. (If I break something, it'll probably show up in multiple test suites. So if the agent sees two failures and 20 successes, maybe it won't bother me, but if it then sees another failure or two it should.)
I am imagining I could look at the change between the prior and posterior probabilities and use that to estimate the entropy of that new test result, and perhaps also compute my personal entropy (if that makes any sense -- as in, the bot should assume I don't know what any of the results are until it notifies me, at which point it should assume I am aware of the current full set of results and not bother me again until enough additional results have come in to substantially change the probability estimate of my change being bad.)
Or something like that. I'm really looking for the right mathematical framework to compute things like this. I know next to nothing about machine learning or... well, mathematics in general.
(The above is simplified; I would additionally like to do crazy things like take into account "labeling", where a third party looks at the test failures and decides whether they are real failures or not. But the time between the failure coming in and when that person labels it for me can also be modeled by a distribution, and I'd like to hold off notifications for interesting results if there's a good chance that this other person may tell me that the result is not interesting after all. But only if that's going to happen "soon". Also, we have an automated system for guessing whether something might be an intermittent vs real failure, and it'd be nice to take its opinion into account as well.)
