4
$\begingroup$

I have historical data with labels and features about medical data about treating cancer. The labels represents:

How happy is the patient after the treatment (other patients did a survey in the past). From scale 1 till 7.

There features are numerical: Via facial recognition we extracted six types emotions on a scale from zero till one.

This is performed for patients who are not able to talk and are in electric wheelchair with a tablet.

I trained a random Forrest Regressor. That predicts how happy will be patient on a scale from 1 till 7.

After the patient is done with the treatment the patient fills in a survey behind a screen. Each patient has a unique user ID. How would you apply personalization via reinforcement learning in order to know happy the patient will be during the next treatment.

The happiness of the patients it's face should be updated if this is not correct for this user.

For example: I want the agent to update the model for this patient it's belief (the old model that we trained) Let's say we predicted 6, but actually it was 4 next time this same patient arrives, we want to estimate the patients happiness with the new updated model.

My model can be correct for patient A, and incorrect for patient B. Because for patient B some emotion was more or less important compared to patient A. How would you solve this problem and it should be scalable for 2000+ patients. Patients with cancer are coming back to the hospital for treatment (on average 10 or 20 treatments). And each patient has an Unique ID.

The model is making the prediction. Based on the predictions the doctors "receive" an happiness estimation.

The model makes: the wrong prediction for patient B but the correct one for patient A. Because patient B indicated after the treatment that the estimation was incorrect based on the scale scores. But Patient A indicated that his prediction was correct.

$\endgroup$
  • 1
    $\begingroup$ Are you certain this is a reinforcement learning (RL) problem? RL usually requires an automated agent that takes actions, and those actions in turn should affect the state. Even if the machine learning here performs some display function, that is not enough to be RL from what I understand of your issue. The only way it would be RL is if the treatment was decided by a computer (and alternative/exploratory treatments would have to be suggested too, in order to find the best ones, which seems a bit scary) $\endgroup$ – Neil Slater Nov 2 '17 at 17:03
  • $\begingroup$ Yes I want the agent to update the model for this patient it's belief (the old model that we trained) Let's say we predicted 6, but actually it was 4 next time this same patient arrives, estimate the patients happiness with the new model $\endgroup$ – Tanja Meeren Nov 2 '17 at 18:09
  • 1
    $\begingroup$ That does not seem like RL. RL would be if the model did something to the patient (or instructed the carers to do something to the patient) and that changed how the patient felt. Instead it looks like you want to refine your predictive model based on new data, because it has made a mistake and you have more information . . . ? $\endgroup$ – Neil Slater Nov 2 '17 at 20:39
  • 1
    $\begingroup$ It is OK there is no need to clarify the question further for me, although you could use edit link to add these details, they might be useful for someone. I was asking about the "reinforcement-learning" tag (and in the title). I do not think RL applies here, and think you should remove it because it confuses the question $\endgroup$ – Neil Slater Nov 2 '17 at 21:05
  • 1
    $\begingroup$ Reinforcement Learning is for making game-playing bots, or making self-driving cars, or making agents that invest money, or control factories etc. RL algorithms learn by doing things and observing what happens (they can also learn from a history of someone or something else doing things, but the goal is generally to learn the best actions to take). What you appear to want is to refine predictions of a Supervised Learning algorithm based on getting more data about a patient. You don't appear to want to write an algorithm that tells the doctors what care to provide? $\endgroup$ – Neil Slater Nov 2 '17 at 21:22
1
$\begingroup$

Let's first begin with a primer to reinforcement learning. Reinforcement Learning is a method of learning that predicts the action to perform at a given state or predicts the value associated with a given state and action pair.

So, Neil is correct that this problem is not explicitly a reinforcement learning problem. This begs the question, how does one define a reinforcement learning problem. It is defined as a 5-tuple containing the following elements. $S$ is the set of states, which must be finite. $A$ is the set of actions available at a given state, it is also sometimes written as $A_s$. $P_a(s, s') = \Pr(s_{t+1}=s' \mid s_t = s, a_t=a)$ is the probability that action $a$ at state $s$ will lead to state $s'$. $R_a(s, s')$ is the reward received when using action $a$ to transition from state $s$ to $s'$. $\gamma$ which is the discount factor which is a value between $0, 1$ inclusive to discount rewards at certain steps.

Now, we have to frame the question in the form of a reinforcement learning problem. You need to establish a state, in your case this would be the information about the patient that you believe has a high correlation with their happiness (or is a causal factor for their emotions). Now, you have a state of your patient with these features you've engineered. Now you must decide how to frame your question appropriately. Here you have two choices, either, you can choose what their emotion is and update your weights (your model weights, or the weights you want to output) as necessary (based on the true labels) or you can use your model to choose an action (in this case it would be a class of 1-7). These approaches are labelled as $Q$ learning or $\pi$ Policy methods. These are the primary methods present in RL, one chooses the value to assign to a given state and action $Q(a, s)$ and the other takes in a state $\pi(s)$ and maps it to an action to perform. You can decide which one you'd like to use based on convenience/experience.

Once this is done, you'd have to decide on an architecture you'd like to use to update the gradients, this is totally up to you. You can do a literature review to see what's going on in this domain and use what you think is appropriate. Just as an example, let's say you'd want to use a Policy method that is updated by a vanilla CNN. You would pass in the state to your CNN in some form that would be interpretable by the CNN and then choose an action based on the class labels. Now, you would have a reward function which would choose how far you are from the true class (you can do this heuristically, like Happiness is 10 units from Sadness). The reward function would be propagated to the network and it would update the weights as required.

PERSONAL COMMENTS: I think this is completely unnecessary for the current task. In my opinion a simple vision classifier should do perfectly well. There are an abundance of datasets that are available which takes in facial features and classifies sentiment/happiness. Also, a lot of statisticians believe that numeric values for happiness/sadness is a bad metric because questions like, "How different is very happy from happy" are hard questions to answer in the context of associating a reward with them. Do take Neil's comments into consideration.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.