4
$\begingroup$

Background
I'm a Data Scientist and am being asked to come up with a set of metrics/KPIs to assess my annual performance, and things like bonuses (and in the worst case being fired) depend on that. And of course I want them to align to my personal Data Scientist growth goals as well. My position entails:

  • working on machine learning projects from the start to end (getting data access, exploration, modelling, results, presentation, helping in productionising etc)
  • Analysing certain topics to give suggestions to the management
  • A part of which could be to educate stakeholders on using new tools to analyze on their own

What I've observed is that the job of a Data Scientist is rather complicated when it comes to measuring success, and things like performance appraisals metrics are then harder to design.

My Intuition:
On one hand, things like number of machine learning models deployed seem like a direct measurable metric, but on the other hand all models are not equal. Some may need a lot of time while others might be ready relatively quicker. The same goes for number of topics analysed/explored, as in exploration the required effort could vary a lot more than in modelling.

The only thing I can clearly put is the cumulative business value of the machine learning models developed, or of the business decision made from an exploration/analysis. This, however, puts the responsibility on the stakeholders a lot, putting an overhead work (to devise this value) needed to approach our team, and the stakeholders might hesitate approaching us. On the other hand, I believe that the stakeholders should be clear about the value of every topic they're working on, and the ones they requested other teams to work on.

Apart from business value, I also think about including something to assess the quality of every modelling project or analysis, something like a quality score, so as to drive me to improve the quality of my approach, to present better etc. But I'm not sure if quantifying the overall quality of such things could be practical and indicative consistently of what it was meant to be.

Question:
What could (generally) be good performance evaluation metrics for such a Data Scientist?
OR
What should the technical approach be to devise such metrics?

$\endgroup$
4
  • 2
    $\begingroup$ there are things, that cannot be measured aliong these lines. For example, supporting colleagues to solve their problems and various improvements made overall that are not concrete enough to be measured, yet took some time away. Time invested to solve a problem, shows dedication and hard-working attitude, but rarely acknowledged as such. $\endgroup$
    – Nikos M.
    Nov 16, 2021 at 20:05
  • $\begingroup$ Maybe pointing out that the solutions already achieved (there are various metrics for those, like accuracy,..), were won through hard work, which many times is under the radar of supervisors. In the final analysis, results are what are visible, no matter the hard work behind them $\endgroup$
    – Nikos M.
    Nov 16, 2021 at 20:08
  • $\begingroup$ One can even state explicitly all the background work that was needed in order to get a model running. Eg, implementation problems that had to be addressed, research needed to narrow down alternatives worth pursuing, all these can be explicitly addressed $\endgroup$
    – Nikos M.
    Nov 16, 2021 at 20:12
  • $\begingroup$ Or maybe they have no idea what they are doing in managing you: press.princeton.edu/books/hardcover/9780691174952/… $\endgroup$
    – Stephen Rauch
    Nov 29, 2021 at 4:31

1 Answer 1

1
$\begingroup$

Most explicitly: did the models increase the top line or help out the bottom line? By what degree? That's really what stakeholders are concerned about in my opinion.

More implicitly: did the work you did allow for the model making process to be streamlined? If so, to what degree? Does the pre-work make a huge difference (obviously yes but you might have to spell it out).

The point of having a data science team is to leverage massive quantities of data to make decisions. How many decisions were made based on your work either wholly or partially?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.