3
$\begingroup$

I would greatly appreciate if you could let me know what to report in the following steps of CRISP-DM?

  • Build Model: what should be reported for parameter settings, models and model description? I used grid search to tune hyperparameters.
  • Assess Model: what should be reported for model assessment and revised parameter settings?
  • Evaluate Results: what should be reported for assessment of data mining results?

enter image description here

In fact, I used just Logistic regression to do a classification task using a procedure like what is depicted below: enter image description here

Could the final model evaluation on test data set be considered as a part of Evaluate Results or it should be done on new data?

|improve this question
$\endgroup$
  • $\begingroup$ hi, I've been busy these days and couldn't respond to your comment. I don't know whether you've solved it or not. I noticed you have deleted your comment. Have you solved it? $\endgroup$ – Media Jun 26 '18 at 6:54
  • $\begingroup$ @Media Hi Thanks a lot. Your welcome. I was in doubt about my solution so, I posted it in the second and third comment of the answer provided to this post. Still I am not sure about it. $\endgroup$ – ebrahimi Jun 26 '18 at 7:53
  • $\begingroup$ I guess it highly depends on your task. for classification tasks reporting the accuracy is customary but for regression tasks, I've even seen plotting learning-curve-like graphs too. $\endgroup$ – Media Jun 28 '18 at 19:46
  • $\begingroup$ @Media Hi. I would greatly appreciate if you could let me know whether model selection: 1- just refers to algorithm selection i.e., finding the best predictive algorithm e.g., SVC vs Logistic Regression? or 2- it could interchangeably used with hyperparameter tuning of a given learning algorithm e.g., SVC? Thanks in advance. Best regards, $\endgroup$ – ebrahimi Jul 6 '18 at 14:30
  • $\begingroup$ Hi, Basically I guess the latter is more correct due to the fact that when ever you try to find an appropriate algorith, you will attempt to set and tune the hyper parameters. $\endgroup$ – Media Jul 6 '18 at 14:47
3
$\begingroup$

The way you are trying to present the outcome is pretty good.

I cannot say that the following procedure is the standard procedure in my scenario I did something like this: CRISP-DM

This is how I presented to my managers to make them understand the procedure which I followed.

I made a slide for each and every segment and highlighted the points which are necessary for them to be known.

While showing the results, I named the data(accuracies) differently

  1. Train Set Accuracy: Which is 70% of the total dataset, derive the accuracy % also called model accuracy
  2. Test Set Accuracy: Test set which is 30% of the total dataset, derive the accuracy called Test Accuracy
  3. Blind Test Accuracy: Which is completely new data, here I trained the model with whole dataset and then tested using the new data(you can also call it as Validation set)

This is how I presented: Accuracy Table

Since yours is also a classification problem to explain it better I gave them the breakdown:

Further Breakdown of Accuracies

Do let me know if you have any issues, would love to help you.

|improve this answer
$\endgroup$
  • $\begingroup$ yeah true, the about is clear right? Do you think anything else needs to be appended? $\endgroup$ – Toros91 Jun 17 '18 at 9:45
  • $\begingroup$ I first divide the whole dataset into training and test data set. The training set will be used to tune hyperparameters via k-fold cross-validation. Then, the whole training data set is trained using the best determined parameters in the previous step. Finally, the model is tested on the test data set. So, I decided to report: $\endgroup$ – ebrahimi Jun 17 '18 at 11:59
  • $\begingroup$ parameter settings: hyperparameter tuning via grid search , models: the results (confusion matrix) of whole training data set, model description: the coefficients of logistic regression, model assessment: the learning curve, revised parameter settings:nothing is needed and assessment of data mining : the results (confusion matrix) of test data set. @Toros91 $\endgroup$ – ebrahimi Jun 17 '18 at 12:03
  • $\begingroup$ This would go in the report right? $\endgroup$ – Toros91 Jun 17 '18 at 12:04
  • $\begingroup$ Yes, I am thinking of doing so. Is it right? $\endgroup$ – ebrahimi Jun 17 '18 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.