In one of my projects I have 7 output classes. However, while this makes up a baseline set of results, I also want to test the impact of some experimental parameters.

To avoid any information disclosure I'll abstract the question details...

Let's assume that we have input data of speed, type of car, etc. and the output variables are the action the car is doing (i.e. braking, turning left/right, etc.). The input variables will be the same for each of the classes in the first instance - for example each car will travel at 50mph.

While it is relatively trivial to assume this will be trained all in one model, the question here is how to best evaluate the impact of changed parameters on classification accuracy. By changed parameters here, I refer to speed as an example. Let's say we want to evaluate the impact of varying speeds (i.e. 50mph as a baseline and then 60, 70, 80, etc) on the classification accuracy.

Would all of this data (as well as the baseline 50mph that we have trained on before) be used to retrain the single model? Or would one use separate datasets on different models and do some sort of cross-evaluation?

It seems sensible to potentially adopt a multi-label-like approach here, however I've read mixed things online and I figured it would be best to get more solid clarification.

  • 1
    $\begingroup$ I don't see how multi-label would help for this. Since your goal is to study the impact of a feature on performance, I guess I would just train the model and then split the test set based on the feature value, e.g. one group of instances with speed in the range 50-60, the 2nd one for the range 60-70, etc. Then these groups of instances can be evaluated independently, so you could plot the relation between the feature and performance. $\endgroup$
    – Erwan
    Jun 7, 2022 at 19:59


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.