1
$\begingroup$

I recently had a phone interview with a consumer tech company for a quant position. The question was basically, "imagine a facebook style social network site. Six months ago a new feature called 'mentions' was added which allows you to tag your friends with an @ sign. How would determine whether this feature was a success?"

I was a bit taken aback by how broad the question was. I first asked if the feature was given to everyone in the network or a sample, to which the interviewer responded "you decide" - meaning I could approach the analysis either way. I talked in general terms about calculating week over week usage of the feature as well as month over month. I also discussed computing a baseline metric for product interaction and then comparing the usage of the new mentions feature relative the baseline statistic. Overall I left the interview feeling quite dumb, as I have a pretty solid command of stats, but came away looking like an idiot.

Are there specific statistical procedures to test for something like this? al la A/B testing, or some kind of hypothesis test? And secondly, is there a good framework for approaching these types of open ended case study style questions in general?

$\endgroup$
  • $\begingroup$ Look at click-through rate of the tagged friends and the spam-mark rate of others, if they are allowed to mark it as such ('coz tagging your friends on random posts spams everyone else). $\endgroup$ – Emre Feb 17 '17 at 22:56
  • $\begingroup$ The correct answer is "see if the company gets more advertising revenue before or after". $\endgroup$ – Spacedman Feb 18 '17 at 8:31
  • $\begingroup$ This is a nice trap. The answer of Paul is the right answer I guess. $\endgroup$ – Robin Feb 18 '17 at 17:20
2
$\begingroup$

This question (something I've asked variants of several times in interviews) has absolutely nothing to do with statistical or other quantitative procedures. What is being asked here is for an understanding of the overall data mining process itself. The first thing to determine is what the definition of success. So you have to ask. The stakeholder usually will not volunteer this unless asked anyway. Then, depending on the answer describe the overall process for data mining based on this end goal.

$\endgroup$
0
$\begingroup$

Personally, I don't think this question is reasonable. The first thing you need to do is determine from the stakeholders what "success" is. This could be increase traffic, increase revenue, etc. Without knowing how the stakeholders view success you can find all kind of interesting things in whatever data you have and never satisfy your clients. It is very common for data scientist to look for a needle in the haystack only to find the wrong needle if they can even find one.

$\endgroup$
0
$\begingroup$

I believe open ended questions like this one have the goal to see what is your thought process. The interviewer wants to know how would you tackle the problem, what do you do first, what hypothesis do you consider, and, most important, how do you defend your decisions. Asking questions and, sometimes, even thinking out loud can be helpful as you show the interviewer your thought process.

I would probably employ a similar approach and assume that this feature was rolled out for a subset of users. Then I would check metrics such as user engagement (here you can define that as number of comments, number of liked comments, number of comment replies, or a function of all of those) and perform an A/B test. Based on the results a conclusion can be reached if this new feature is indeed a success (based on the aforementioned metrics) or not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.