What makes you confident in your results? At what point do you think you can present your work to tech illiterate superiors?

Question

I understand that the models are only as good as the data you get, and bad design can generate really bad data. Nonrandom sampling, unbalanced/incomplete designs, confounding, can make data analysis really hard.

At what point should one be confident that they ran a useful model? Do you just do a cross-validation with a training/test dataset and call it a day? Obviously "all models are wrong, some are useful" but at some point the tradeoffs with excluding too many parameters by LASSOing and strange transformations by getting BIC down become glaring.

tl;dr at the end of the day what makes you go "I did the right thing for my company/project, and this should work"

Answer 1

Hey Welcome to the Site!

What you are saying is right, Data Science din't reach to the stage where it has some standard methods for achieving this(standard procedures, don't know we would be able to reach that stage in near future). But we have some general standards like:

1. Forecasting: ETS,ARIMA,SARIMA etc
2. Prediction: Linear Regression,Random Forest, GLM, Neural Network etc
3. Classification: Logistic Regression, Random Forest etc

When you go to granular level it is hard to generalize, as every business problem is different and one single method cannot be used for solving all the business problems.

So, to answer the next question how do you get confidence that the outcome is good enough, I assume that you have heard about RMSE,MAPE and many more for predictions and confusion matrix for classification problem. We use these metrics to see access the models performance, for example if you are trying to classify whether the given cell is cancer cell or not, there are 100 records in which 90 are non-cancer cells and 10 are cancer cell, your model gives 99% accuracy but could classify 5 out of 9 literally 55% of the total in such scenarios you need to look cannot use accuracy, you need to use F1 score etc. As you were asking about a model right, all models are not useful. True not all models built are going to go for production level, you would choose the best one and productionize it. You can re-train your model on a basis(Daily, Weekly,Monthly based on business requirement). Would you call it a day off post completion of validation? I wouldn't, I would go to the Subject Matter Expert present him the results ask him/her for their insights, if they both are inline then I would do a Beta testing on some actual data and then productionize it.

Now to address your last question, There is no standard saying that this is good or bad, if it works for you, your Business then that is a Good Model. To convenience your mangers and subject matter(Data) experts, you need to dig deep into the data try all different scenarios ask as many questions as possible. Try understanding the data very well. So, you can answer Business Questions with data supportive answers(This is possible only when you are well-worse with data). As they are very good with business they would be asking questions with respect to business, you need to be ready with all such scenarios by understanding business and data well.

Finally, I do have a feeling like you do. I did alot of things but nothing worked but you shouldn't be unhappy as you understood that these are the ways which would lead you to Unsuccessful results(best example is Thomas Alva Edison has used 1000 diff metals before using Tungsten to make a bulb). Similarly all the methods which we have tried are different steps you have tried to get the solution. My funda is, did I try something different/new everyday or not. Crucial part of this process is, maintenance of clear documentation at each and every step. Which would come in handy in the near future.

Anything in R&D is never a waste it is just an other try or experiment, so your work is never waste. Your are trying to build a solid base for the bright future of your company.

Answer 2

What makes you confident in your results?

The appropriate method to evaluate whether you have modeled a real signal or noise is completely dependent on the question you are asking and the modeling approach you've used to address it. Many very thick books have been written on this topic, often constraining their attention to one problem domain and/or type of model. The complexities associated with model evaluation are a big component of why data scientists generally have graduate degrees. Which brings us to the second part of your question:

At what point do you think you can present your work to tech illiterate superiors?

Your tech illiterate superiors don't have that graduate degree that would inform them how to evaluate your analysis. They're trusting you to present honest and accurate results. It is extremely easy to mislead people who aren't stats-fluent into believing whatever narrative you want to present. It's your responsibility to make sure your results are air tight, or at least in synch with your client's risk tolerance.

Your results are ready to be shared when you are satisfied that you are interpreting them correctly, and you have a plan for how to communicate them plainly.

at the end of the day what makes you go "I did the right thing for my company/project, and this should work"

1. I have built a model that accomplishes my goal.
2. I am satisfied that my modeling approach is sound and repeatable. If I'm working on a prediction task, I also want to be sure that my model generalizes well to out-of-sample data.
3. I have evaluated the projected impact of utilizing my model and am reasonably confident that the benefits of applying it justify the time and effort I am putting in building it.
4. I have a clear path forward to implementation. I have a technical plan for making my results actionable, and know whose support I need to make it happen.
5. I am confident that I can communicate my results in a way that will convince non-technical stakeholders that my results are real and assuage their concerns.