How to approach a Data Science case study question?

Question

I recently had a phone interview with a consumer tech company for a quant position. The question was basically, "imagine a facebook style social network site. Six months ago a new feature called 'mentions' was added which allows you to tag your friends with an @ sign. How would determine whether this feature was a success?"

I was a bit taken aback by how broad the question was. I first asked if the feature was given to everyone in the network or a sample, to which the interviewer responded "you decide" - meaning I could approach the analysis either way. I talked in general terms about calculating week over week usage of the feature as well as month over month. I also discussed computing a baseline metric for product interaction and then comparing the usage of the new mentions feature relative the baseline statistic. Overall I left the interview feeling quite dumb, as I have a pretty solid command of stats, but came away looking like an idiot.

Are there specific statistical procedures to test for something like this? al la A/B testing, or some kind of hypothesis test? And secondly, is there a good framework for approaching these types of open ended case study style questions in general?

Answer 1

This question (something I've asked variants of several times in interviews) has absolutely nothing to do with statistical or other quantitative procedures. What is being asked here is for an understanding of the overall data mining process itself. The first thing to determine is what the definition of success. So you have to ask. The stakeholder usually will not volunteer this unless asked anyway. Then, depending on the answer describe the overall process for data mining based on this end goal.

Answer 2

Personally, I don't think this question is reasonable. The first thing you need to do is determine from the stakeholders what "success" is. This could be increase traffic, increase revenue, etc. Without knowing how the stakeholders view success you can find all kind of interesting things in whatever data you have and never satisfy your clients. It is very common for data scientist to look for a needle in the haystack only to find the wrong needle if they can even find one.

Answer 3

I believe open ended questions like this one have the goal to see what is your thought process. The interviewer wants to know how would you tackle the problem, what do you do first, what hypothesis do you consider, and, most important, how do you defend your decisions. Asking questions and, sometimes, even thinking out loud can be helpful as you show the interviewer your thought process.

I would probably employ a similar approach and assume that this feature was rolled out for a subset of users. Then I would check metrics such as user engagement (here you can define that as number of comments, number of liked comments, number of comment replies, or a function of all of those) and perform an A/B test. Based on the results a conclusion can be reached if this new feature is indeed a success (based on the aforementioned metrics) or not.