Measuring Value in Data Science?

Question

Why does it seem that it's difficult to find out how people in data science create measurable value? All I find on the internet are buzzwords like data cleaning, visualization, and writing about the data. This is equivalent of describing a landscaper as trimming trees and grass or saying an investment banker spreads M&A. That is insufficient. Let my type out an insufficient answer:

A data scientist uses database querying languages like SQL as well as statistical programming languages like R to design experiments with data and visualizing such experimental conclusions to decision-makers in the hopes of making better decisions and thus creating value for a company/cause.

How is this insufficient? Well, how do we know that data scientists' recommendations actually create value? If we can't measure the value of our analyses, then how can we determine if hundreds of hours of learning/working in the subject is meaningful? Extreme example:

George is a data scientist working for Company X. After hundreds of hours of data cleaning/experimentation, he concludes Decision A will benefit Company X. George convinces a Product Manager to apply Decision A that ultimately increases the revenues of Company X by $0.01

How do we know that several years of work may not amount to basically nothing? Let me give you a depressing and real/plausible example:

Instead of becoming a data scientist, George becomes a portfolio manager, managing investment portfolios for clients. After 20 years of management, George has a track record of -1% per annum, compared to an S&P 500 return of 8%.

In the above example, George is a useless person and has lost value for people. How do data scientists know that they aren't destroying value, and if we can't figure that out, what are the steps to avoid value destruction and create the most value for companies?

Answer 1

I read your question as:

tl;dr. How do you know that a data scientist adds value to a company?

The job of a data scientist is to pick out the best data set, scrub it, fit a statistical model. And most importantly, a data scientist needs to know the problem statement and why is the analysis being done.

For example, consider a DS team in an e-commerce company. Their primary (let's assume) is to build a highly effective recommender system, which needs to adapt itself to a lot of variables, like the user behaviour, the session-based behaviour, user segments, etc.

So, if the recommender system is still not being able to get the company a considerable CTR or upsells/cross-sells, then maybe the team is not adding proper value to the company.

So, a data scientist needs to have a keen knowledge of the business value which an experiment is expected to add to the company. The decisions made from the experiments should ultimately add value to a company (monetary or otherwise). If not, then just like any other employee, it means that the person have failed in the job.