21
$\begingroup$

With all the hoopla around Data Science, Machine Learning, and all the success stories around, there are a lot of both justified, as well as overinflated, expectations from Data Scientists and their predictive models.

My question to practicing Statisticians, Machine Learning experts, and Data Scientists is - how do you manage expectations from the businesspeople in you company, particularly with regards to predictive accuracy of models? To put it trivially, if your best model can only achieve 90% accuracy, and upper management expects nothing less than 99%, how do you handle situations like these?

$\endgroup$
  • 1
    $\begingroup$ Nice question! But should be a community wiki I suppose $\endgroup$ – Alexey Grigorev Jun 15 '15 at 12:00
  • $\begingroup$ Nice question. Been there, done(and doing) that :D $\endgroup$ – Dawny33 Nov 30 '15 at 11:12
11
$\begingroup$

Rational business people don't pay for accuracy, they pay to either

  • save money on a profitable process (thereby making it more profitable), or by
  • creating new money (creating new profitable processes).

So any project that is undertaken has to be couched in terms that reflect this. The first step is always understanding which of the two processes you are working on, and you should have a clear idea of how that might be achieved, while keeping in mind that as you make progress the details of how you do that might change.

If you can improve the accuracy of a process, you can probably make money for the firm and the business people will invest in your progress. The only rational reason a business person could have for insisting on 99 percent accuracy and reject 90 percent is if they already had a way of doing it that was better than the 90 percent. If that is the case they are of course justified in their position.

Understanding and presenting the business case of the projects you are working on in terms that the Business People understand is part of the maturing process of any Engineer. It is not unique to Data Science at all, though Data Science has some unique aspects (like lower maturity but higher probability of fortuous serendipity discovery - at least in today's environment.

A relevant process that is close to Data Science that makes this step explicit can be found here:https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining

But most Enterprise Architecture Frameworks are similarly applicable.

$\endgroup$
  • 1
    $\begingroup$ Nicely done. Your answer kind of demarcates expectation from reality. Bravo! $\endgroup$ – untitledprogrammer Jun 16 '15 at 19:45
  • $\begingroup$ Oddly the upvote here did not register in my points. $\endgroup$ – Mike Wise Jun 17 '15 at 6:08
10
$\begingroup$

Gather competitive counterparts. Try and determine a state-of-the-art and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven models are not created statically, they develop dynamically because a good scientist will always try to find ways to improve it.

Upper management personnel should know that a data scientist explores new methods, sometimes/often without knowing their quality. They should know that machine learning techniques do not produce perfect models right away. If they did, it wouldn't be challenging anyway.

A data scientist ought to be evaluated by how he justifies and discusses his results and how he plans the future. A way for the management personnel to handle with their expectations is to not have unrealistically high ones.

Still, if reasonable results are expected in the field of context, think about these questions:

  1. Did/will results get better over time?
  2. Are future expectations positive?
  3. How well are results compared to similar systems (from competitors)?
$\endgroup$
5
$\begingroup$

I like this question because it gets at the politics that exist in every organization. In my view and to a significant degree, expectations about model performance are a function of the org culture and degree to which an organization is "technically literate." One way to make clear what I mean is to consider the differences between the 4 big "data science" entities -- Google, FB, Amazon and Yahoo -- versus the 4 big agency holding entities -- WPP, Omnicon, Interbrand and Publicis. Google, et al, are very technically literate. The agencies, on the other hand, are known to lean towards tech phobia. What's the evidence for this? First off, the technically literate group was founded or are run by engineers, computer scientists, geeks and people with strong tech backgrounds. Who runs the tech illiterate companies? Marketers who have risen to prominence by virtue of their soft communication and people skills. And not only that, having worked in some of these shops in NYC, I can testify that these organizations systematically punish and/or push out the highly technically literate types as not a "fit" with the culture. Next, consider their aggregate (stock) market caps, The tech literate group adds up to about 800 billion dollars while the tech illiterate group amounts to 80 billion. Tech literate entities are 10x bigger than the others in market cap. This is a clear statement of the market's expectations and it's not high for the illiterates. So, by extrapolation, what kind of hope can you have for challenging the "predictive accuracy" expectations of bozos like these?

So, given that cultural breakout and depending on where you fall, you should have more or less realistic expectations. Of course, different "tech illiterate" entities will have managers who know what they're doing, but for the most part, these entities are dominated by the idiocy of the lowest common denominator in tech skills, i.e., people who are at best technical semi-literates (and dangerous) or, more commonly, totally innumerate but don't know it. Case in point, I worked for a guy who wanted words like "correlation" scrubbed from c-suite decks. This is an extreme case: after all, every secretary knows what a "correlation" is.

This raises the issue of how one deals with the maddeningly naive and innumerate when they ask a really dumb question like, "Why aren't you getting 99% predictive accuracy?" One good response is to reply with a question like, "Why would you assume such an unrealistically high PA is even possible?" Another might be, "Because if I actually got 99% PA, I would have assumed that I was doing something wrong." Which is highly likely to be true, even with 90% PA.

The there's the more fundamental question of the insistence on PA as the sole criterion for model value. The late Leo Breiman left many footprints on the statistical and predictive modeling community of which PA is one. His primary concern with PA was to address the many criticisms being made in the 90s regarding the instability and error inherent in running a single CART tree. His solution was to motivate “random forests” as an approximate and provisional method that would maximize accuracy and reduce instability by eliminating tree structure. He benchmarked the lower MSE from ~1,000 iterative RF “mini-models” against the error from a single logistic regression model. The only problem was that he never bothered to mention the glaring apples to oranges comparison: what might the logistic regression MSE have been if it had also been performed 1,000 times?

The 2008 Netflix Prize offered a sizeable monetary reward to any statistician or team able to improve upon the MSE of their recommender system. At the time, Netflix was spending $150 million a year on this system, convinced that the costs were more than recovered in customer loyalty and purchase of movies that would otherwise never have been chosen. The eventual winners used a complex ensemble of 107 different models.

As Netflix learned however, the real problem was that, from a fully loaded cost perspective, the actual improvement in error over their current model was a mere 0.005% reduction in the 5 point ratings. Not to mention that the IT costs in time, heavy-lifting and maintenance of the winning ensemble of 107 models more than nullified any gains from the error reduction. Given this, Netflix eventually abandoned the pursuit of MSE and no more Netflix Prizes have been awarded

And this is the point: minimizing predictive error can be easily gamed or p-hacked and is prone to analyst fraud (i.e., finding a solution that glorifies the analyst’s modeling skills, positively impacting his potential end-of-year bonus). Moreover, it is a completely statistical solution and goal set in an economic and business vacuum. The metric provides little or no consideration of ancillary, collateral costs -- the very real operational consequences evaluated from A to Z that should be an integral part of any fully-loaded, trade-off based decision-making process.

This has become one of those issues that is embedded in organizations and is very, very difficult to change. In other words, I am fully aware that I am tilting at windmills with this rant about the caveats with the use of PA.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.