From the Montana article "Kinematics of Contact and Grasp", if I have a ball roll on the plane without sliding, the motion equation is described below: \begin{equation*} \begin{bmatrix} \dot{u}_{2} \\ \psi \end{bmatrix} = \begin{bmatrix} M^{-1}_{2}R_{\psi}\\ T_{1}+T_{2}R_{\psi} \end{bmatrix} M_{1}\dot{u}_{1} \end{equation*}

${(1)}$ for Ball, ${(2)}$ for Plane. ${\psi}$ is the angle of contact between ${x}$ axis of coordinates of ball and plane. ${K, T, M}$ are the curvature form, torsion and metric tensor at time $t$ relative to coordinates of ball and plane.

We have \begin{equation*} R_{\psi} = \begin{bmatrix} \cos_{\psi} -\sin_{\psi}\\ -\sin_{\psi} -\cos_{\psi} \end{bmatrix} \end{equation*}

How can I use the Bellman equation by Sutton and Barto to discrete path planning of the motion of the ball?

\begin{align} v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s'} P_{ss'}^a(R_{ss'}^a + \gamma v_{\pi}(s')) \end{align}

From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from one point to another point.

The goal is how to optimize the path planning of the point in the motion.

I do not know how to determine the policy $\pi,$ and how to build up a function from Bellman equation. Please leave comments.


1 Answer 1


From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from 1 point to another point.

That's already a good start. Still, I would recommend to be a bit more concrete to clear everything up. The action is not only the rolling ball, but has to be some kind of instructive control, e.g. move the ball 1cm forward, steer 10° clockwise, accellerate by a - just to name a few possibilities. Given this Action set $A$ with $a\in A$. $P_{ss\prime}^a$ models the state transition probabilities - if you are in a deterministic environment, it is constant 1 for one state and 0 for all others. $R$ is the reward, and here is where it becomes tricky. You want to find the best trajectory to the goal, thus you want to penalize if the agent takes unneccessary actions, leading to the agent reaching the goal as fast as possible. I would recommend a constant -1 for that purpose, if you end the episode when the ball reaches the goal. If this is not the case, it has to be more sophisticated, e.g. the negative distance of the ball towards the goal. This should be everything you need to get your hands dirty.

  • $\begingroup$ @André The ball rolls from A to B via a motion equation. If the trajectory is a curve, I still confuse how to formulate the math model using the Bellman equation via discretization. For example, which part of the equation which can be optimized. $\endgroup$ Sep 3, 2018 at 13:59
  • $\begingroup$ @LAMNGOCTAM Thanks for the additional info. On a curved plane, things get more complicated. Your $P_{ss\prime}^a$ is the probably the most interesting part here. I would suggest you discretize the plane in a fine grid, this will already make things easier. If you now have a cell in this grid, you can calculate how the ball moves in one timestep. You can also apply an action on this movement and calculate its influence. And you will have to incorporate that into $P_{ss\prime}^a$. Make it 1 for the state you just calculated and 0 for the rest. $\endgroup$
    – André
    Sep 3, 2018 at 14:14
  • $\begingroup$ Thank you. I am trying to establish the motion planning on discrete time (on the grid) then using the part of Bellman equation to optimize the path planning. The previous motion equation is used for continuous time. $\endgroup$ Sep 6, 2018 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.