Optimization motion planning by using Bellman equation

Question

From the Montana article "Kinematics of Contact and Grasp", if I have a ball roll on the plane without sliding, the motion equation is described below: \begin{equation*} \begin{bmatrix} \dot{u}_{2} \\ \psi \end{bmatrix} = \begin{bmatrix} M^{-1}_{2}R_{\psi}\\ T_{1}+T_{2}R_{\psi} \end{bmatrix} M_{1}\dot{u}_{1} \end{equation*}

${(1)}$ for Ball, ${(2)}$ for Plane. ${\psi}$ is the angle of contact between ${x}$ axis of coordinates of ball and plane. ${K, T, M}$ are the curvature form, torsion and metric tensor at time $t$ relative to coordinates of ball and plane.

We have \begin{equation*} R_{\psi} = \begin{bmatrix} \cos_{\psi} -\sin_{\psi}\\ -\sin_{\psi} -\cos_{\psi} \end{bmatrix} \end{equation*}

How can I use the Bellman equation by Sutton and Barto to discrete path planning of the motion of the ball?

\begin{align} v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s'} P_{ss'}^a(R_{ss'}^a + \gamma v_{\pi}(s')) \end{align}

From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from one point to another point.

The goal is how to optimize the path planning of the point in the motion.

I do not know how to determine the policy $\pi,$ and how to build up a function from Bellman equation. Please leave comments.

Answer 1

From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from 1 point to another point.

That's already a good start. Still, I would recommend to be a bit more concrete to clear everything up. The action is not only the rolling ball, but has to be some kind of instructive control, e.g. move the ball 1cm forward, steer 10° clockwise, accellerate by a - just to name a few possibilities. Given this Action set $A$ with $a\in A$. $P_{ss\prime}^a$ models the state transition probabilities - if you are in a deterministic environment, it is constant 1 for one state and 0 for all others. $R$ is the reward, and here is where it becomes tricky. You want to find the best trajectory to the goal, thus you want to penalize if the agent takes unneccessary actions, leading to the agent reaching the goal as fast as possible. I would recommend a constant -1 for that purpose, if you end the episode when the ball reaches the goal. If this is not the case, it has to be more sophisticated, e.g. the negative distance of the ball towards the goal. This should be everything you need to get your hands dirty.