I need to design a controller that tries to collect as much falling objects (raindrops) from an unknown source. Beneficial to solving the problem could be the recognition of the seemingly random generation of the raindrops.

y  ^
   | 0 0 1 0 0 0 0
   | 0 0 1 0 0 1 0
   | 1 0 0 0 0 1 0
   | 0 0 0 1 1 0 0  

So, I have a binary vector signal (raindrop generated yes/no over 4 positions) over discrete time steps. I can only see the raindrops when they are 10 steps away from hitting the ground. The basket has a capacity of one raindrop. The control actions available are step left, step right or do nothing. Any source of pattern recognition could help me design a more advanced controller. Are there methods that can recognize periodic patterns in this type of data?

  • $\begingroup$ am only allowed to use the "visible" raindrops as an information source. Visible means that they are in a bin of Y and T positions (y and t on the graph), so for example 4 positions and 10 time steps in advance. Only then am I allowed to use the position to optimize over. But I am allowed learn a pattern over visible raindrops. $\endgroup$
    – hasdrubal
    Jun 26, 2017 at 9:15
  • $\begingroup$ The time steps at which I can see them is limited, so decisions have to be made for a more advanced controller. If there is a robust method to filter periodic patterns, I will use it. However, from the answers I doubt if there is one. $\endgroup$
    – hasdrubal
    Jun 26, 2017 at 20:22
  • $\begingroup$ All right. So the input gives me the complete list of all raindrops in the complete simulation. But I am only allowed to use the raindrops that are 10 steps away from hitting the ground. These are the visible ones. $\endgroup$
    – hasdrubal
    Jun 26, 2017 at 21:10
  • $\begingroup$ The control action is pretty simple. Step left, step right or do nothing. The basket can take one raindrop at the time when it hits the ground. $\endgroup$
    – hasdrubal
    Jun 26, 2017 at 21:11

2 Answers 2


If it is really close to random (background noise, bits from random number generators or hash codes), I don't expect there is much you can do, except create a large collector. (Or are you only allowed one collector?)

You may want to throw in a few random checks. Those won't prove anything but may give you some insight into the pattern.

If you want to create a successful model, you'd have to come up with some assumptions about parameters that could affect the success of your model, and create and test that model.

For instance, one of my assumptions would be that there are rows with higher probability. One other assumption would be that there is some hysteresis effect (either positive or negative), so the probability may depend on the actual outcome of the previous event(s).

  • $\begingroup$ I only have one collector, wouldn't be much of a challenge if I had a big one. And you are right: if the signal is close to random, I wouldn't bother modeling it. I don't know how I can measure this, hence the question. I will analyse the probability of each row, it might prove usefull. I don't know if there is any hysteris in the generator, so that's a good point as well. But for now I need tools to analyse patterns in the data, because I don't have the background in this particular area. $\endgroup$
    – hasdrubal
    Jun 25, 2017 at 18:18

There is unlikely to be any useful pattern analysis for this problem.

I cannot prove it, but I think it highly likely that the raindrops are being generated using a pseudo-random process. Even if they are not, you have been given more than enough information in what your controller can "sense" in this simulation, so that predicting future raindrops is not necessary.

Instead, this is a problem of optimal control and planning. In machine learning, often reinforcement learning can be used to solve problems like this - potentially Q-Learning would work here, but it would be quite hard to implement it so that it learned quickly due to the large number of states. You would need to use a function approximator like a neural network, and although that can be made to work nicely here, it does not seem necessary.

I don't think you need any Data Science technique here, this is closer to more traditional AI planning. However, there are strong links between planning and learning models, so what I am going to suggest is also something you might see in game-playing systems that use a combination of machine learning (reinforcement learning) and planning.

A quick analysis shows that you have a large number of visible states ($2^{42}$) which would take a long time to produce optimal rules for using the simplest reinforcement learning algorithms - although it is feasible. However, you also have perfect knowledge of the dynamics and a fully deterministic system within what you can see. In addition, the branching factor of your action decisions is not high, just 3 (and sometimes 2) per step - so for instance looking 8 steps ahead will involve checking under 10,000 scenarios. That seems possible to do on every time step, which immediately suggests a simple search-based planning algorithm - similar to Monte Carlo Tree Search, except in your case you can just brute-force all combinations up to an arbitrary horizon. As long as the horizon is far enough ahead, you can achieve optimal results this way, because it only takes 3 steps to fully traverse the different positions the collector can be in.

  • On each time step:

    • Observe the current state and position of incoming raindrops

    • One by one, generate feasible chains of actions, working through all possibilities, some number of steps ahead (I'm suggesting 8 steps, but wonder if you can get away with less, e.g. just 5).

      • If the sequence of actions would take the controller outside of the allowed area, discard it.

      • Score the sequence of actions by counting how many raindrops you predict it will collect, based on the information about raindrops at each timestep.

      • Keep a record of the best sequence of actions so far.

    • Using the best sequence of actions so far, choose the first action in that sequence, and take that action "for real" in the simulation, advancing by one time step.

    • Discard the rest of the sequence, even though your controller really might take that path, ready for next phase of planning. In the simplest cases you don't revise the plan using clever tree pruning techniques, you just brute force another search after each update. A more advanced algorithm may be able to preserve the most likely sequences to go forward to next time step, and save computation (at the cost of complexity and memory used to store candidates).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.