1
$\begingroup$

Is there any good reference for methodologies on quality assessment of machine learning algorithms design. The scope is wide and embraces each steps from data preparation to generalisability testing.

Further, are there any automated "stress-test tools" for ensuring that the algorithms is not biased, that it fits the alledged performances?

$\endgroup$
2
$\begingroup$

Here's a few fragments of information relevant to your question.

  • For generalisability testing, it is most common to split the dataset on which you aim to apply a Machine Learning model into a training, validation and testing set. K-Fold cross validation is also common: splitting the dataset into training and validation sets multiple times, randomly, to ensure a good performance on the validation set isn't just pure luck.
  • Likewise, to ensure your algorithm is not biased and fits alleged performances, there is no better way than to benchmark it on test datasets that the model has not seen during training time. Some deep learning frameworks come with tools that generate reports regarding your model's performance, such as Tensorflow's Tensorboard.
  • To find out what attributes influence an algorithm, there are various approaches you can take. Interpretability tools such as LIME will help you see what characteristics of the data you trained your model on affect your model's predictions most.

    This article on interpretability continues the discussion, goes more in-depth, presents a few frameworks as well as a tutorial you can follow.

$\endgroup$
1
  • $\begingroup$ thank you,I know about these tools. I was looking for something more in depth and on the methodology side $\endgroup$
    – epsilones
    Mar 7 '19 at 9:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.