# Neural network architecture for multinomial logit model

I'm working on a neural network model to predict the outcomes of horse races. To date, I've built a model in R using a multinomial logit model (similar to a logit model but with N outcomes, where N = no. of horses in a race). I've also carefully read Andrew Task's excellent Grokking Deep Learning book and learned the basics of PyTorch. I've now got to the point where I'm able to build a very simple neural network using only starting prices (i.e., odds for horses at start of race) as input and the net correctly works out that it should bet on the favourite. I'm using the architecture of 16 inputs (odds for up to 16 runners, set to zero if fewer than 16 runners in a race), 16 outputs (probability of horse winning a race), 1 hidden layer with \sqrt{16 \ times 16} nodes, a RELU activation function applied to the hidden layer, and a SOFTMAX activation function applied to the output layer. I apply argmax on the output layer to chose the winning horse.

Based on my earlier analysis in R, betting on the favourite results in a win rate of 35.7% whereas betting on the horse chosen by my multinomial logit model results in a (lower) win rate of 23.4%, i.e., for now my model underperforms backing the favourite. I've been able to replicate the 35.7% figure using the neural network with the architecture described above (actually I undershoot this figure but I know how to change the architecture to exactly hit this figure). Surprisingly, however, when I swap out market price (which wouldn't really be available ahead of a race for betting purposes) and swap in exactly the same features I used in the multinomial logit model I manage to achieve a win rate of only about 17%, even if I train the model with 500 epochs.

As I'm relatively knew to the world of neural networks, I've no idea how to go about tweaking the architecture or hyperparameters of the neural network to improve its performance such that it's at least able to match the performance of the classical statistical model I built earlier. (I'm making the bold assumption that a neural network should be able to do at least as well as a classical statistical model, provided the net is architected correctly.) Any pointers would be greatly appreciated! (FYI, this is a personal project to help me learn deep learning, and not any commercial enterprise.)

In the plot below, confused-dust and absurd-universe refer to versions of the model with market prices as sole inputs whereas comic-wood and true-resonance refer to versions using the same set of features as in the multinomial model.

Thank you!

• Have you tried using multiple layers of neurons in the network? In theory this would allow the network to create more complicated features which can help the network to better learn the underlying relationships within the data. Apr 21 at 14:59

I eventually arrived at the solution below, which can be used to replicate the result of a multinomial logit regression:

class ParsLin(nn.Module):
""" Parsimonious version of Linear """
def __init__(self, input_layer_nodes, output_layer_nodes):
super().__init__()

# Check if output_layer_nodes is an integer multiple of input_layer_nodes
if input_layer_nodes % output_layer_nodes != 0:
raise ValueError("inputt_layer_nodes must be an integer multiple of output_layer_nodes")

self.input_size = input_layer_nodes
self.output_size = output_layer_nodes

self.coefficient_size = input_layer_nodes // output_layer_nodes

weights = torch.zeros(self.coefficient_size)
self.weights = nn.Parameter(weights)  # nn.Parameter is a Tensor that's a module parameter.

# Xavier (Glorot) initialization
nn.init.xavier_uniform_(self.weights.view(1, self.coefficient_size))

def forward(self, x):
# Reshape races tensor to separate features and horses
n = x.shape[0]
reshaped_input = x.view(n, self.coefficient_size, self.output_size)

# Transpose tensor to have each horse's features together
transposed_input = reshaped_input.transpose(1, 2)

# Multiply transposed tensor with coefficients tensor (broadcasted along last dimension)
marginal_utilities = transposed_input * self.weights

# Sum multiplied tensor along last dimension
utilities = marginal_utilities.sum(dim=-1)

return utilities

class MLR(nn.Module):
""" Parsimonious version of LinSoft intended to replicate a multinomial logit regression with
alternative specific variables and generic coefficients only """
def __init__(self, input_layer_nodes, output_layer_nodes, bias=None): # bias is unused argument and will be ignored
super().__init__()

self.neural_network = nn.Sequential(
ParsLin(input_layer_nodes, output_layer_nodes),
nn.Softmax(dim=1)
)

def forward(self, x):
logits = self.neural_network(x)
return logits

• Continuous variables should be standardized also. That makes a difference. Apr 27 at 14:00