How can the deviance of my model be higher than the null deviance?

Question

I have a simple logistic regression model in Python, set up using sklearn. The code for training the model (and calculating some metrics across multiple runs) looks something like this:

for i in range(30):
    X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.75)
    train_scaler, test_scaler = StandardScaler(), StandardScaler()
    X_train, X_test = train_scaler.fit_transform(X_train), test_scaler.fit_transform(X_test)

    logistic = LogisticRegression(solver='newton-cg', class_weight=weights)
    logistic.fit(X_train, y_train)
    
    y_test = np.array(y_test)
    predicted = logistic.predict(X_test)
    accuracies = np.concatenate((accuracies, np.array([[sum((predicted==y_test)*(y_test==i))/sum(y_test==i) for i in range(2)]])))
    
for i in range(2):
    print(f"The accuracy for category {i} is {np.mean(accuracies[1:, :], axis = 0)[i]}.")

And the output looks a bit like this:

The accuracy for category 0 is 0.6437247830550223. The accuracy for category 1 is 0.5607368278690966.

This model is fit on 408 examples with 306 predictors. Admittedly, this not much data relative to the number of predictors, but it still achieves an average accuracy of around 61% on the test set, predicting about 64% of label 1 and 56% of label 2 correctly, so better than chance. By contrast, a model that simply selects each class with a probability equal to its representation, the null model, hovers around 50%, unsurprisingly.

However, I calculate the deviance using the labels and variables for the test set as:

2*sklearn.metrics.log_loss(y_test, model.predict_proba(X_test), normalize=False)

In the case of the second model, instead of model.predict_proba(X_test), we have an array that looks like [[1-p, p], ..., [1-p, p]], since the probability of each label is constant in the null model, looking something like this:

array([[0.52205882, 0.47794118],
       [0.52205882, 0.47794118],
       [0.52205882, 0.47794118],
       [0.52205882, 0.47794118]
       ...
       [0.52205882, 0.47794118]])                
         

This value is higher (about twice as higher) for the first model than for the second! Even when I write out my own explicit formula for the deviance, nothing changes. To my understanding, in theory, the (absolute value of the) deviance of any model should be strictly lower than that of the null model, since higher deviance indicates greater difference from the saturated model.

How can the deviance of a fitted logistic model be higher than that of the null model, particularly when the logistic model shows better predictive accuracy?

---

Note: the data is private, so I cannot share it here.

Answer 1

After looking over the model, I discovered that the problem was insufficient convergence on the test set due to overfitting, which was resolved through increasing the strength of the L2 regularization.

The deviance of the fitted model is guaranteed to be lower than that of the null model, but only if it has properly converged on the universe of data and is evaluated on the optimal $y$. Otherwise, it is possible for an overfit model to produce a deviance (evaluated on the test set) that is lower than that of the null model (evaluated using the average value from the test set for the probability, and thus converged on the test set), even if it shows better-than-chance performance.

In particular, since the deviance for a binary logistic model consists of the sum of terms of the form $y_k\ln(p(1,k))+(1-y_k)\ln(1-p(1,k))$, which incorporates the magnitude of the probabilities, it is possible for most of the terms in this sum to be lower in absolute than the terms in the sum for the null model, with the overall sum still being more negative due to outliers, as was the case here. In other words, if when the model is good, it is moderately good, but when it is bad, it is quite bad, this behavior can occur.

These outliers seemed to be a symptom of overfitting, unsurprisingly given the large number of variables relative to the samples (perhaps 75% as many variables as samples), as is common in textual analysis. To solve this overfitting, I substantially increased the regularization strength, which resolved the problem and made the deviance of the fitted model lower in absolute value.