Reinforcement Learning different patients

Question

I have historical data with labels and features about medical data about treating cancer. The labels represents:

How happy is the patient after the treatment (other patients did a survey in the past). From scale 1 till 7.

There features are numerical: Via facial recognition we extracted six types emotions on a scale from zero till one.

This is performed for patients who are not able to talk and are in electric wheelchair with a tablet.

I trained a random Forrest Regressor. That predicts how happy will be patient on a scale from 1 till 7.

After the patient is done with the treatment the patient fills in a survey behind a screen. Each patient has a unique user ID. How would you apply personalization via reinforcement learning in order to know happy the patient will be during the next treatment.

The happiness of the patients it's face should be updated if this is not correct for this user.

For example: I want the agent to update the model for this patient it's belief (the old model that we trained) Let's say we predicted 6, but actually it was 4 next time this same patient arrives, we want to estimate the patients happiness with the new updated model.

My model can be correct for patient A, and incorrect for patient B. Because for patient B some emotion was more or less important compared to patient A. How would you solve this problem and it should be scalable for 2000+ patients. Patients with cancer are coming back to the hospital for treatment (on average 10 or 20 treatments). And each patient has an Unique ID.

The model is making the prediction. Based on the predictions the doctors "receive" an happiness estimation.

The model makes: the wrong prediction for patient B but the correct one for patient A. Because patient B indicated after the treatment that the estimation was incorrect based on the scale scores. But Patient A indicated that his prediction was correct.

Answer 1

Let's first begin with a primer to reinforcement learning. Reinforcement Learning is a method of learning that predicts the action to perform at a given state or predicts the value associated with a given state and action pair.

So, Neil is correct that this problem is not explicitly a reinforcement learning problem. This begs the question, how does one define a reinforcement learning problem. It is defined as a 5-tuple containing the following elements. $S$ is the set of states, which must be finite. $A$ is the set of actions available at a given state, it is also sometimes written as $A_s$. $P_a(s, s') = \Pr(s_{t+1}=s' \mid s_t = s, a_t=a)$ is the probability that action $a$ at state $s$ will lead to state $s'$. $R_a(s, s')$ is the reward received when using action $a$ to transition from state $s$ to $s'$. $\gamma$ which is the discount factor which is a value between $0, 1$ inclusive to discount rewards at certain steps.

Now, we have to frame the question in the form of a reinforcement learning problem. You need to establish a state, in your case this would be the information about the patient that you believe has a high correlation with their happiness (or is a causal factor for their emotions). Now, you have a state of your patient with these features you've engineered. Now you must decide how to frame your question appropriately. Here you have two choices, either, you can choose what their emotion is and update your weights (your model weights, or the weights you want to output) as necessary (based on the true labels) or you can use your model to choose an action (in this case it would be a class of 1-7). These approaches are labelled as $Q$ learning or $\pi$ Policy methods. These are the primary methods present in RL, one chooses the value to assign to a given state and action $Q(a, s)$ and the other takes in a state $\pi(s)$ and maps it to an action to perform. You can decide which one you'd like to use based on convenience/experience.

Once this is done, you'd have to decide on an architecture you'd like to use to update the gradients, this is totally up to you. You can do a literature review to see what's going on in this domain and use what you think is appropriate. Just as an example, let's say you'd want to use a Policy method that is updated by a vanilla CNN. You would pass in the state to your CNN in some form that would be interpretable by the CNN and then choose an action based on the class labels. Now, you would have a reward function which would choose how far you are from the true class (you can do this heuristically, like Happiness is 10 units from Sadness). The reward function would be propagated to the network and it would update the weights as required.

PERSONAL COMMENTS: I think this is completely unnecessary for the current task. In my opinion a simple vision classifier should do perfectly well. There are an abundance of datasets that are available which takes in facial features and classifies sentiment/happiness. Also, a lot of statisticians believe that numeric values for happiness/sadness is a bad metric because questions like, "How different is very happy from happy" are hard questions to answer in the context of associating a reward with them. Do take Neil's comments into consideration.