What to report in the build model, asses model and evaluate results steps of CRISP-DM?

Question

I would greatly appreciate if you could let me know what to report in the following steps of CRISP-DM?

Build Model: what should be reported for parameter settings, models and model description? I used grid search to tune hyperparameters.
Assess Model: what should be reported for model assessment and revised parameter settings?
Evaluate Results: what should be reported for assessment of data mining results?

In fact, I used just Logistic regression to do a classification task using a procedure like what is depicted below:

Could the final model evaluation on test data set be considered as a part of Evaluate Results or it should be done on new data?

hi, I've been busy these days and couldn't respond to your comment. I don't know whether you've solved it or not. I noticed you have deleted your comment. Have you solved it? — Green Falcon, Jun 26, 2018 at 6:54
@Media Hi Thanks a lot. Your welcome. I was in doubt about my solution so, I posted it in the second and third comment of the answer provided to this post. Still I am not sure about it. — ebrahimi, Jun 26, 2018 at 7:53
I guess it highly depends on your task. for classification tasks reporting the accuracy is customary but for regression tasks, I've even seen plotting learning-curve-like graphs too. — Green Falcon, Jun 28, 2018 at 19:46
@Media Hi. I would greatly appreciate if you could let me know whether model selection: 1- just refers to algorithm selection i.e., finding the best predictive algorithm e.g., SVC vs Logistic Regression? or 2- it could interchangeably used with hyperparameter tuning of a given learning algorithm e.g., SVC? Thanks in advance. Best regards, — ebrahimi, Jul 6, 2018 at 14:30
Hi, Basically I guess the latter is more correct due to the fact that when ever you try to find an appropriate algorith, you will attempt to set and tune the hyper parameters. — Green Falcon, Jul 6, 2018 at 14:47

Toros91 · Accepted Answer · 2018-06-17 09:07:47Z

3

The way you are trying to present the outcome is pretty good.

I cannot say that the following procedure is the standard procedure in my scenario I did something like this:

This is how I presented to my managers to make them understand the procedure which I followed.

I made a slide for each and every segment and highlighted the points which are necessary for them to be known.

While showing the results, I named the data(accuracies) differently

Train Set Accuracy: Which is 70% of the total dataset, derive the accuracy % also called model accuracy
Test Set Accuracy: Test set which is 30% of the total dataset, derive the accuracy called Test Accuracy
Blind Test Accuracy: Which is completely new data, here I trained the model with whole dataset and then tested using the new data(you can also call it as Validation set)

This is how I presented:

Since yours is also a classification problem to explain it better I gave them the breakdown:

Do let me know if you have any issues, would love to help you.

answered Jun 17, 2018 at 9:07

Toros91

2,3373 gold badges13 silver badges31 bronze badges

$\begingroup$ yeah true, the about is clear right? Do you think anything else needs to be appended? $\endgroup$
– Toros91
Jun 17, 2018 at 9:45
$\begingroup$ I first divide the whole dataset into training and test data set. The training set will be used to tune hyperparameters via k-fold cross-validation. Then, the whole training data set is trained using the best determined parameters in the previous step. Finally, the model is tested on the test data set. So, I decided to report: $\endgroup$
– ebrahimi
Jun 17, 2018 at 11:59
$\begingroup$ parameter settings: hyperparameter tuning via grid search , models: the results (confusion matrix) of whole training data set, model description: the coefficients of logistic regression, model assessment: the learning curve, revised parameter settings:nothing is needed and assessment of data mining : the results (confusion matrix) of test data set. @Toros91 $\endgroup$
– ebrahimi
Jun 17, 2018 at 12:03
$\begingroup$ This would go in the report right? $\endgroup$
– Toros91
Jun 17, 2018 at 12:04
$\begingroup$ Yes, I am thinking of doing so. Is it right? $\endgroup$
– ebrahimi
Jun 17, 2018 at 12:12

| Show 8 more comments

1 Answer 1