6
$\begingroup$

I work in an analytical role at a a large financial services firm. We do a ton of daily reporting over metrics that rarely change in a meaningful way from day to day. From this daily reporting, our management is required to extract what was important yesterday and what important trends have developed / are developing over time.

I want to change this to a model of daily exception reporting and weekly trend reporting.

Features might include:

  • User report consolidation (so there's only one daily email)
  • report ordering based upon level of variance from past performance (see the most important stuff first)
  • HTML email support (with my audience, pretty counts)
  • Web interface to allow preference changes, including LDAP support (make administration easier)
  • Unsubscribe feature at the report level

Here's what I'd like to know:

  • What are the practical problems I might run into?
  • What is the best way to display the new reports?
  • How should I define an "exception"? How can I know if my definition is a good one?
  • I assume I'd be using a mix of Python, SQL, and powershell. Anything else I should consider, e.g. R? What are some good resources?
Improve this question
$\endgroup$
3
  • $\begingroup$ I think this is just about programming or architecture? I don't see a requirement for engineering+statistics or things people would call ML. I would have said StackOverflow is the best place, but if you can narrow this down to more than just asking for product recommendations. $\endgroup$
    – Sean Owen
    Oct 26, 2014 at 20:14
  • $\begingroup$ I just made some edits that added questions that are more specific. I think that this question is on topic because once you've done your analyses, you're going to have to communicate the results to others, and that is the gist of Brandon's question. $\endgroup$
    – JenSCDC
    Oct 29, 2014 at 16:02
  • $\begingroup$ I think the question is clearly too wide: consider how many answers there is to "What are the practical problems I might run into?". In the end, the collection of all answers to that question is the system you want to create. :) IMHO, you should narrow this down to "How should I define an exception? How can I know if my definition is a good one?"; it is the most related to Data Science, the others are for Programmers SE. $\endgroup$
    – logc
    Jan 8, 2015 at 21:20

4 Answers 4

Reset to default
5
$\begingroup$

Try exploring the rich field of "Anomaly Detection in Time Series". Control charts and CUSUMs (or cumulative sum control charts) might help you.

Simple Bullet Graphs might be all you need. Based on historical data and domain knowledge, define normal variance. Then make it clear to stakeholders when the current value is outside of predefined ranges.

Stephen Few is an expert in business dashboards. Any of his books will help you.

If you are open to R, try Shiny to create simple interactive web applications (It is very straightforward). There is also an open source package for anomaly detection, both local and global.

Create quick prototypes and get feedback!

Improve this answer
$\endgroup$
3
$\begingroup$

The optimal solution very much depends on multiple factors, including your (current and future) business and IT processes, stakeholders' needs and preferences, and, in general, business and IT architectures. So, IMHO it's difficult to answer this wide and not well-defined question. Having said that, I hope that you will find the following earlier answers of mine here on Data Science StackExchange relevant and useful.

Improve this answer
$\endgroup$
3
$\begingroup$

I think this point is the core of your question:

  • How should I define an "exception"? How can I know if my definition is a good one?

I would go about it as follows:

  • Go back to the managers who receive the current reporting and ask them to give you examples of reported exceptions that were actually exceptions, i.e. they were acted upon and the action yielded some benefit.
  • Ask them also if there were any features in that report that were not plainly viewable, but led them to think it was an exception, e.g. the month-on-month difference was not reported, but it could be computed from the week-on-week difference.

Treat the examples from the first as labels for training; you want to learn "what is an exception" and you need to have an expert answer that question for you.

Treat the suggested features as new features for your classification.

If you cannot get the experts to answer the questions you make, then try to infer them from their interaction with your reports: how many of those reports were downloaded, how many times are they mentioned, how are they ever used in decision-making ... Try to separate the important ones from the uninteresting ones, and there you have your labels.

Improve this answer
$\endgroup$
1
$\begingroup$

Here's some practical advice from my own experiences-

  1. The first thing to do is to convince management the the change will be for the better. A mock-up of a sample report can be very useful here.

  2. Even if management agrees, they'll still want to know why variances have occurred, and hence you'll need to be able to supply more data.

  3. Although exception reporting is best, management will want to see everything anyway, as doing so makes them feel as though they are doing something useful.

  4. Don't change everything at once- too big a change at once can cause resistance.

  5. For how to best present the data, read Edward Tufte's books, at the very least his first one, "The Visual Display of Quantitative Data".

  6. Defining what's an exception can be hard, because the recipients will each have their own ideas. Using say, a 95% confidence interval is good, but it won't be universally liked. Some people will consider any change above $X significant, and others will want to see everything that's more than X% different from the prior period. Have fun with this part :(

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.