Are there any control-flow/conditional statements in AI/ML models?

Question

I was recently asked this during an interview.

When we write a C program, it has a control-flow in the form of conditional statements like if, while, for that executes a condition, and that decides how the control flows through the application.

In AI/ML models, e.g. CNNs like ResNet, I don't see any control-flow (either static or dynamic). It's just a series of consecutive convolution layers. So,

• where are these control-flow/conditional statements in AI domain? What do they look like?
• Which models have them? Any examples?

Answer 1

Yes, there are control flow constructions in neural networks, for instance:

• Variable-length sequence classification with recurrent units: in text classification tasks with recurrent units (vanilla RNNs, LSTMs, GRUs), both at training and inference time, you basically iterate over the time dimension of the input batch, passing the next token and the previous steps' state to the recurrent unit. This is a for loop.

• Variable-length sequence generation: in text-generation encoder-decoder networks, at inference time the decoder generates one discrete token at a time until a special "end-of-sequence" token is generated. This is pretty much a while construction. The decoder can be based on any type of neural net units: LSTMs, CNNs, self-attention (Transformer), etc.

As you can imagine, this kind of control flow construction is typical of variable-size input/outputs. You won't see this kind of construction in most image processing architectures.

Control-flow stuff is more difficult to implement in declarative deep learning frameworks, like Theano or Tensorflow 1.x. In Tensorflow 1.x, there was actually a specific primitive to implemente while loops: tf.while_loop. With imperative frameworks like Pytorch or Tensorflow 2.x, control-flow constructions are far easier to implement, because you can integrate normal logic between the computations.