In my office, we recently built an AI model for project success prediction using binary classification.

Though the dataset size was small (977 records), my boss still wanted to go ahead with the POC because they wanted to know how AI works and get a feel of the workflow and reception from business users. (For these reasons, he wanted me to go ahead with POC despite me advising against it due to low data size).

Now with the model being built and AI predictions (binary classification) were validated with the project managers and it was found to be correct in 80% of the cases.

I highlighted the limitations of small data size and told him that if we have to scale it, make it work like a machinery (like automated pipeline), it's better and worth only if we do with large dataset.

So, my questions are below

a) these project sucess predicted resulted in revenue for company (like 70-100k USD). Our biz users are so busy and they are tackling many projects. So, they had no visibility of the projects that AI identified and they had no idea it will be success (or any revenue will come from them). So, definitely AI helped bring in additional revenue of 70-100. Though my manager acknowledges and gives credit to AI, as a data scientist, am concerned and not happy due to small data size (and feel like it was luck). He wants to call this POC a succes but am not sure. Should I accept the credit given by my manager for AI prediction or whay should I do now? Can I check with experts here on what would you do if you were in my situation dealing with a POC of a snall dataset

Note - Both my company, my boss are new to data science. Am 1 year experienced in solving problems using classical ML algorithms

  • $\begingroup$ If you think the performance was luck, maybe try bootstrapping or repeated CV. Small data in the right model and/or with a sufficiently easy problem can still give great lift. $\endgroup$
    – Ben Reiniger
    Sep 15 at 15:09
  • $\begingroup$ I'd say setting a Baseline and a CV is part of the modelling approach. Maybe you could have done this to ensure stability and ability of AI to be better than baseline. But honestly here I am not sure there is a problem. Yeah it is often a bit annying when AI is oversold. I usually try to speak about Supervised Leanring or Machine Learning. But other than that you should probably accept the credit. This way around is way better than the contrary (not getting credit for usefull work). $\endgroup$
    – lcrmorin
    Sep 18 at 8:17
  • $\begingroup$ @Icrmorin - Yes, understand. I did baseline (using few models) and also did CV. But my problem is currently there is no non-AI solution in the 1st place (to compare AI performance against it)? Users follow up with orders based on their business experience etc but no formalized/systematic approach to track projects. They basically follow-up on projects which has high revenue potential etc. So, through AI we just try to came up with logical assumptions and made this kind of work. So, would like to seek experts opinion here. Hope this info helps further to share your additional insights if any $\endgroup$
    – The Great
    Sep 18 at 10:15