I recently had a phone interview with a consumer tech company for a quant position. The question was basically, "imagine a facebook style social network site. Six months ago a new feature called 'mentions' was added which allows you to tag your friends with an @ sign. How would determine whether this feature was a success?"
I was a bit taken aback by how broad the question was. I first asked if the feature was given to everyone in the network or a sample, to which the interviewer responded "you decide" - meaning I could approach the analysis either way. I talked in general terms about calculating week over week usage of the feature as well as month over month. I also discussed computing a baseline metric for product interaction and then comparing the usage of the new mentions feature relative the baseline statistic. Overall I left the interview feeling quite dumb, as I have a pretty solid command of stats, but came away looking like an idiot.
Are there specific statistical procedures to test for something like this? al la A/B testing, or some kind of hypothesis test? And secondly, is there a good framework for approaching these types of open ended case study style questions in general?