In one of my projects I have 7 output classes. However, while this makes up a baseline set of results, I also want to test the impact of some experimental parameters.
To avoid any information disclosure I'll abstract the question details...
Let's assume that we have input data of speed, type of car, etc. and the output variables are the action the car is doing (i.e. braking, turning left/right, etc.). The input variables will be the same for each of the classes in the first instance - for example each car will travel at 50mph.
While it is relatively trivial to assume this will be trained all in one model, the question here is how to best evaluate the impact of changed parameters on classification accuracy. By changed parameters here, I refer to speed as an example. Let's say we want to evaluate the impact of varying speeds (i.e. 50mph as a baseline and then 60, 70, 80, etc) on the classification accuracy.
Would all of this data (as well as the baseline 50mph that we have trained on before) be used to retrain the single model? Or would one use separate datasets on different models and do some sort of cross-evaluation?
It seems sensible to potentially adopt a multi-label-like approach here, however I've read mixed things online and I figured it would be best to get more solid clarification.
