How to show prototype of output before building model

Question

Currently in my work, we are working on a POC for a AI project.

We intend to do a binary classification using traditional classification algorithms.

However, my boss wants me to show a feel of the output/taste of how the output would be like.

So, am here seeking suggestions on how can it be done?

If my original data is iris dataset, I can show some examples from online website. But for my real data, how can I show such visualizations? Lime explainer etc, (without even building model).

He wants to see how the output would be like before I can even build model

How do you all do this at you work?

Answer 1

You could simulate data and fit a model to it as if it were real data. there are packages and functions in R and Python to do this. You'd have to be very clear that the data is faked. You could then examine the model and produce graphs as if it were a real one.

This has the downside that it involves writing all the code and writing code to sim data, which can take time, so best to do this if the project is highly likely to go ahead.

The upside is that it gives a good idea to your stakeholder of what to expect and if they greenlight the project then you have most of the code written.

Answer 2

Provided you have the data already, and the data is labelled (i.e., split into the two classes $A$ and $B$), it makes sense to produce a number of visualisations to gauge what the model output would be.

If you start with traditional classification algorithms like logistic regression, then the model output is going to be the probability of belonging to a particular class (say class $A$).

Given the model output is a probability, it would make sense to look at the proportion of observations that fall into class A by several factors that are available on your dataset. For example, you could look at the average proportion that fall into class A over time, by age, by height, etc.

EDIT

There's two parts to 'explaining the model without building it'. You need to

(1) Show what sort of predictions you're expecting, and
(2) Gauge which factors are going to drive that prediction.

Say you're building a model to determine whether or not it will rain in the next week ($Y$). Let $y$ be a realisation of $Y$ with $y=1$ when it rains, and $y=0$ otherwise. Let $X$ be a vector of factors you're going to use to determine if it rains or not.

On (1), take a given factor from $X$ (call it $x_i$). If $x_i$ is continuous, bin it into several groups ($x_{i,j}$ $j = 1, 2, ..., g$ where $g$ is the number of groups). These groups should align to the business understanding. In the absence of business understanding, just split the variable into deciles (or similar). For each group $j$ of $x_i$, compute

$$\delta_{i,j} =\frac{\sum_{l=1}^ny_i}{n}$$

where $n$ is the number of records on the dataset. Repeat this for each of the $g$ groups. After this, plot the results. The x-axis will be the $x_{i,j}$'s and the y-axis will be $\delta_{i,j}$'s. By doing this, you can determine what sort of predictions the model will produce.

On (2), you should look into the information value.