Neural network architecture for multinomial logit model

Question

I'm working on a neural network model to predict the outcomes of horse races. To date, I've built a model in R using a multinomial logit model (similar to a logit model but with N outcomes, where N = no. of horses in a race). I've also carefully read Andrew Task's excellent Grokking Deep Learning book and learned the basics of PyTorch. I've now got to the point where I'm able to build a very simple neural network using only starting prices (i.e., odds for horses at start of race) as input and the net correctly works out that it should bet on the favourite. I'm using the architecture of 16 inputs (odds for up to 16 runners, set to zero if fewer than 16 runners in a race), 16 outputs (probability of horse winning a race), 1 hidden layer with \sqrt{16 \ times 16} nodes, a RELU activation function applied to the hidden layer, and a SOFTMAX activation function applied to the output layer. I apply argmax on the output layer to chose the winning horse.

Based on my earlier analysis in R, betting on the favourite results in a win rate of 35.7% whereas betting on the horse chosen by my multinomial logit model results in a (lower) win rate of 23.4%, i.e., for now my model underperforms backing the favourite. I've been able to replicate the 35.7% figure using the neural network with the architecture described above (actually I undershoot this figure but I know how to change the architecture to exactly hit this figure). Surprisingly, however, when I swap out market price (which wouldn't really be available ahead of a race for betting purposes) and swap in exactly the same features I used in the multinomial logit model I manage to achieve a win rate of only about 17%, even if I train the model with 500 epochs.

As I'm relatively knew to the world of neural networks, I've no idea how to go about tweaking the architecture or hyperparameters of the neural network to improve its performance such that it's at least able to match the performance of the classical statistical model I built earlier. (I'm making the bold assumption that a neural network should be able to do at least as well as a classical statistical model, provided the net is architected correctly.) Any pointers would be greatly appreciated! (FYI, this is a personal project to help me learn deep learning, and not any commercial enterprise.)

In the plot below, confused-dust and absurd-universe refer to versions of the model with market prices as sole inputs whereas comic-wood and true-resonance refer to versions using the same set of features as in the multinomial model.

Thank you!

Answer 1

I eventually arrived at the solution below, which can be used to replicate the result of a multinomial logit regression:

class ParsLin(nn.Module):
""" Parsimonious version of Linear """
def __init__(self, input_layer_nodes, output_layer_nodes):
    super().__init__()
    
    # Check if output_layer_nodes is an integer multiple of input_layer_nodes
    if input_layer_nodes % output_layer_nodes != 0:
        raise ValueError("inputt_layer_nodes must be an integer multiple of output_layer_nodes")
    
    self.input_size = input_layer_nodes
    self.output_size = output_layer_nodes
    
    self.coefficient_size = input_layer_nodes // output_layer_nodes

    weights = torch.zeros(self.coefficient_size)
    self.weights = nn.Parameter(weights)  # nn.Parameter is a Tensor that's a module parameter.

    # Xavier (Glorot) initialization
    nn.init.xavier_uniform_(self.weights.view(1, self.coefficient_size))

def forward(self, x):
    # Reshape races tensor to separate features and horses
    n = x.shape[0]
    reshaped_input = x.view(n, self.coefficient_size, self.output_size)

    # Transpose tensor to have each horse's features together
    transposed_input = reshaped_input.transpose(1, 2)

    # Multiply transposed tensor with coefficients tensor (broadcasted along last dimension)
    marginal_utilities = transposed_input * self.weights

    # Sum multiplied tensor along last dimension
    utilities = marginal_utilities.sum(dim=-1)
    
    return utilities

class MLR(nn.Module):
    """ Parsimonious version of LinSoft intended to replicate a multinomial logit regression with
    alternative specific variables and generic coefficients only """
    def __init__(self, input_layer_nodes, output_layer_nodes, bias=None): # bias is unused argument and will be ignored
        super().__init__()

        self.neural_network = nn.Sequential(
            ParsLin(input_layer_nodes, output_layer_nodes),
            nn.Softmax(dim=1)
        )

    def forward(self, x):
        logits = self.neural_network(x)
        return logits