What are interleaved layers of convolutions?

Question

In the FaceNet paper, they describe the Zeiler&Fergus model like this:

[...] the Zeiler&Fergus model which consists of multiple interleaved layers of convolutions, [...]

What do they mean by interleaved? How does that compare to the inception layers?

Especially, as the Zeiler&Fergus paper states

We use standard fully supervised convnet models throughout the paper, as defined by (LeCun et al., 1989) and (Krizhevsky et al., 2012).

[...]

The top few layers of the network are conventional fully-connected networks and the final layer is a softmax classifier.

Answer 1

The Oxford dictionary explains "interleave" as

Place something between the layers of (something)

or in the context of telecommunication, as

Mix (digital signals) by alternating between them.

In the FaceNet paper, Schroff et. al describe their first architecture as

multiple interleaved layers of convolutions, non-linear activations, local response normalizations, and max pooling layers

which means they create a CNN by "altering between" these layer types (e.g. Conv -> Pool -> Conv -> Pool -> ..). Or with the wording of the first definition, you "place one layer" of each type "between" the other existing layers (i.e. start with a couple of Conv layers, then place Pool layer between two Conv layers, and so on). The final architecture would look somewhat like this:

Conv -> ReLu -> LRN -> Pool -> Conv -> ReLu -> LRN -> Pool -> Conv -> ...

So, this "interleaved" CNN is a network where the different layer types are applied in series, layer-by-layer. As a comparison, in "inception" type CNN, you apply the different layer types in parallel (the graphic below is 1 layer):

              +---> Conv 3x3 ---+
              |                 |
prev layer ---+---> Conv 5x5 ---+---> next layer
              |                 |
              +---> Pool 3x3 ---+