Predicting a relative distribution between the available instances

Question

I am working on a model that is supposed to predict how a given order volume distributes over the available articles in a retail scenario. For simplicity's sake, let's say I'm a retailer that buys apples from different farmers and suppliers, and sells them to supermarkets. I don't have all apples at all times, so there's a set of available apple types at any given time. Based on what I can currently offer, and a given predicted volume for apples in general, I need to predict how the volume distributes among my currently available apple sorts.

I have a list of orders with information on time, article, pricing and a few other meta columns.

My current approach is to aggregate these orders by day and article, then construct a binary dataframe to indicate availability, enriched with pricing and discount features.

I then construct a label frame that contains the relative distribution, normalized so they sum to 1:

I'm training a custom neural network that learns these distributions, but i'm struggling with noise and overfitting.

My typical prediction looks like this:

The model doesn't learn to actually predict 0 for articles that aren't currently available. There's always a significant noise floor that weakens the predictions for the actually available articles. I have looked into a SoftMax layer to clean up the predictions and custom loss functions that punish unavailable articles harder, but I'm honestly a bit stuck.

I'm looking for tweaks I can make to my current approach, and also completely different approaches, since I can't find a lot of info on challenges like this.

Answer 1

Softmax cannot give you zero probabilities, by definition. If this non-zero probability mass is a problem for your predictions, you may want to look into SparseMax, which is a sparse variation of softmax that can actually assign zero to part of the probability space.

It was introduced in the article From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification (published at ICML'16). You can find implementations in both Pytorch and Tensorflow on github.

It may be interesting to check out the recent research of the first author, André F.T. Martins (see his Google Scholar profile, who has been working on sparse neural networks.

Answer 2

One option is to use label smoothing. Re-encode the target labels using this formula:

$$ \text{new_labels}= \text{(original_labels)} * (1 – \text{label_smoothing}) + (\text{label_smoothing} / \text{num_classes})$$

Where label_smoothing is a hyperparameter. If label_smoothing was chosen to be 0.3, the labels of [0, 1, 2] would become [0.1, 0.8, 1.5].

The model might better learn the target values of 0.1, compared to 0.