Is addition of a random control variable a good idea, to ensure that results are correct?

Question

I'm dealing with a complex dataset with many patients who have a condition, and various qualities about these patients. I'm trying to determine patient outcome based on patient qualities.

I'm using xgboost Classifier to try to determine one of the patient outcomes.

In order to ensure that my results are making sense, I've added a random variable, rand which is a random number within (0,10) that will have no impact on patient outcome.

The rand variable should show last in variable importance.

rand is meant as a control variable.

Is adding a control variable like rand a valid method to ascertain that my model makes sense?

Answer 1

This approach makes some sense but it's not the best approach for several reasons.

First, this control variable might not always be last in importance because it's possible that some other variable also don't have any impact at all on the target variable (outcome).

More importantly, the concept of a control group/control variable is useful in cases where one cannot evaluate objectively the effect of a method. Typically a drug trial needs a control group because of the placebo effect and various other biases. The control group plays the role of a baseline, and the performance of the method is measured relatively to this baseline.

In supervised classification, there is a much more direct way to objectively evaluate the effect of the method: this is exactly what we do when we evaluate the performance on a proper test set (fresh instances). Additionally there are simple ways to compare the performance of a system to a baseline: if there is no previous similar system for the task, one can simply use a random baseline or a majority baseline classifier.