Recommender Model for Human Action in Income Protection

Question

Problem Domain I'm working on a project that involves building a model to provide recommendations on the next best step for Human supervisors to take on income protection claims.

Income protection is an area of insurance that involves providing the insured party with a proportion of their income if ever they fall ill or cannot work. It's a safety net designed to help you if you ever get sick or fall on hard times. Many of our customers get Cancer, Depression, Broken bones, etc. and avail of the insurance service, which is usually provided by their company.

On our side, we have a number of human supervisors that manage the ongoing claims. This means scheduling doctor's appointments, getting up to date medical records and assessments/diagnoses, arranging rehab, talking with the customer, etc. (the cost of all of this is also covered under their insurance). These management items are finite, categorical and described as actions.

The supervisor's job is to help the customer Return To Work (RTW) as soon as possible, both to help the customer heal/get back on their feet, and to reduce costs incurred in handling the claim for the business.

I have datasets that describe what actions each supervisor has taken for every claim for the past 20 years (time series/ sequential data), along with text data that describes the progress of the claim, and finally categorical data describing the claim and customer (type of disease, age, occupation, etc.)

What I'm trying to build is a recommender system which harnesses the data to recommend which action maximises the probability of a Return To Work for a given customer in an on going claim.

What I've Tried

Using the data described above, I have built a neural network to predict the outcome of a closed claim. The NN is a combination of LSTM and Dense layers - LSTM for the sequential action data, Dense for the categorical data and Dense for the TfIdf transformation of the text data. There are 39 possible outcomes, one of which is Return to Work, and the network performs relatively well, achieving ~70% accuracy.

I had planned to use the finished model like so: Iterate through all possible actions, and for each action append it to the claim in question to produce a new "hypothetical" action sequence. For each hypothetical sequence do a feed forward through the trained model and see which action maximises the "score" at the index of the output vector which corresponds to return to work. The action that maximises this score is then the recommendation. A simple toy example for an ongoing claim with 10 actions is shown below:

scores = []
# ongoing_claim.shape = (10, 30)
for action in potential_actions:
    hypothesis = ongoing_claim.append(action)
    # ongoing_claim.shape = (11, 30)
    hypothesis = pad_sequence(hypothesis)
    output = model.predict(hypothesis)
    scores[action] = output[index_for_return_to_work]

The recommendation would be the key corresponding to the highest value in scores.

Unfortunately this doesn't work. No matter which action, when I feed forward the new action sequences in the model the scores stay constant for Return to Work - no variation whatsoever. I think this is because simply adding a tiny new data-point considering the amount of data used to train the model does not add enough variance to perturb the output. I have tried using both sigmoid and softmax activation functions on the final layer but this didn't work either.

Question

What kind of model should I use which harnesses all of the data to then examine an ongoing claim and recommend the action which maximises the probability of a particular outcome (in this case Return to Work)?

Answer 1

It looks like the inverse reinforcement learning problem defined by Stuart Russell as

Given

1. measurements of an agent’s behaviour over time, in a variety of circumstances.
2. measurements of the sensory inputs to that agent;
3. a model of the physical environment (including the agent’s body).

Determine the reward function that the agent is optimizing.

It is particularly difficult in your case because you only have observations of a tiny subset of the actual state of a customer and the environment.

I don't think it is solvable without being able to actively explore the actions or without really strong assumptions or a model of the dynamics.

If you can assume that the historical supervisors did their best to help their clients to return to work, your task simplifies to: given the historical actions up to a given point, predict the next historical action.

Even with this simpler problem, you still have to be careful with conditioning, the possible non-stationarity in time, choosing a good model (perhaps similar to what you tried), etc.