1
$\begingroup$

I posted here already but it is marked to close, so thought of posting it here (as this might be the right forum)

Am working on a simple logistic regression with 1000 records and 28 features.

My business users suggest that they want to first see what the AI can do by itself based on our data as it is. Meaning, they don't want me to do feature engineering, trying out multiple algorithms etc.

They want me to avoid all that because they feel it takes time to do feature engineering and they wish to showcase something quicker and earlier. For the 1st cut, they wish to go live with baseline model with no feature engineering (even if it is 50% accuracy).

They are okay with low recall like 30% or 40% (at least for now) for one of the classes because currently there is nothing done to solve this problem. No one is tackling this problem or even thought to solve this problem. So, this is new to them... So, even if it is low recall for negative class, they feel it is something good for them to start (because positive class has high recall). Meaning, they identify those positive cases accurately and go follow up with them. Since this model is reliable (for them) in terms of positive cases, they wish to go live with this.(and focus on those positive cases) Of course, recall for negative cases is a serious concern for them. But at least they have a solution for one of the classes and they are happy. But ultimately, they would like to have solution for negative classes as well. So they suggest me do feature engineering, model experimentation etc after going live. By live, I mean just a simple static dashboard (and not high end MLops etc).

Later, they want to know with all these experiments of model and new features, is recall for negative class is improved or not?

Is this a right way to go further? As a novice data scientist, I don't feel right about this. If it had been at least 80% (my random choice), I would have been bit okay. I don't have any evidence to proof that 80% is the right choice rather than just saying higher no of actuals are predicted correctly.

So my questions are

a) what should I do and what are the pitfalls/points that I should make sure to keep them aware?

b) Is there anything important that I should highlight them?

c) Should this project still be dropped if business is okay to be with 50% acc? Can we continue to use this model as long as business is fine with it?

d) Any real time experience from your model deployment decisions?

Can share your views on this? Would really be helpful for me to learn and also keep them aware?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I completely agree with the answer on statsSE, I don't have much to add to it:

  • Essentially this is a business decision: you can voice your concerns if you think that the company is making a bad decision, but at the end of the day this is their choice to make.
  • There's one point in particular that I think is worth making clear to the company, it's what it means practically for this system to have low performance. For example it's easy to do a random classifier or a classifier which always predicts the majority class, but it's not useful. Are they aware that in the worst case, "low performance" means that the classifier might be as bad as these?
  • Assuming that things are clear for the company, in general I think it's always a good idea to start on a new problem with a basic model which can be used later as a baseline for future improvements. So I don't see any issue with this decision, as long as they understand what it means. As a side note, starting from a very poor baseline model will make your future models look even better ;)

(note that I'm in academia so I don't have any direct experience like this)

Improve this answer
$\endgroup$
10
  • $\begingroup$ can you help me understand this in layman terms? Are they aware that in the worst case, "low performance" means that the classifier might be as bad as these?? $\endgroup$
    – The Great
    Jan 29, 2022 at 9:42
  • $\begingroup$ @TheGreat what I meant is that numerical performance values, like 50% accuracy, are not always easy to interpret for non-experts. For example some people might imagine that 50% accuracy means "half-good", but sometimes a classifier gets 50% just by always predicting the majority class, and this is useless. However I think that most people would understand for instance: "the predictions made by this classifier are not much better than a coin flip" (random baseline for a binary classifier). I don't have any specific idea about this, it's just about avoiding misunderstandings. $\endgroup$
    – Erwan
    Jan 29, 2022 at 22:19
  • $\begingroup$ @TheGreat sorry I don't know Lime, I can't help about this. $\endgroup$
    – Erwan
    Feb 1, 2022 at 19:25
  • $\begingroup$ @TheGreat sorry I don't know Lime, I don't understand what the output represents. It looks like a logistic regression formula but I have no idea what are the variables. $\endgroup$
    – Erwan
    Feb 8, 2022 at 0:08
  • $\begingroup$ @TheGreat yes, but you know it too since you mentioned it in the question ;) In a linear model y=a x +b, the intercept is b. It's the value of y where x is zero. But I'm pretty sure that the problem is not to understand the intercept in general, it's to understand what it means in the context of this output, right? Do you know what are the variables $x_i$ and $y$? If I understand this I may be able to understand the logic. Btw there are some "exp" everywhere in the formula so it's probably related to logistic regression, not linear regression. $\endgroup$
    – Erwan
    Feb 8, 2022 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.