How does “ Sparsity of connections” in CNNs causes the network to have less parameters?

Question

I am studying Andrew NG's lectures on Convolutional Neural Network and he had provided two reasons for CNNs having less parameters compared to Non-Convolutional networks . They are :

1. Parameter Sharing
2. Sparsity of connections .

While I can make sense of the first reason causing CNN to have less parameters . I don't understand why Sparsity of connections , that is , " Each output in a layer comes from a small number of inputs " cause the network to have less parameters .

Isn't the second reason a bit redundant ?

Can someone please explain ?

Answer 1

In fact these involve different aspects of parameters in a CNN.

1. Parameter sharing, means one parameter may be shared by more than one input/connection. So this reduces total amount of independent parameters. Parameters shared are non-zero.
2. Sparsity of connections means that some parameters are simply missing (ie are zero), nothing to do with sharing same non-zero parameter. Parameters in this case are zero, ignored. That means that not necessarily all (potential) inputs to a layer are actually connected to that layer, only some of them, rest are ignored. Thus sparsity of connections.

Answer 2

Andrew Ng is making this point in comparison to a simple Neural network.

Let's say you have a 10x10 image,
In a dense neural network,
- We will connect every 100 neurons to the 100 in the next layer.(Dense)
- Over that, each all will have a distinct weight (No sharing)
So, total parm = 10K

In a Convolution Neural Network, the approach is as shown in this image,

Now,
Weight sharing - The kernel will have the same weight for each pixel in the next layer i.e. it will not have distinct 9 weights for each slide.

Sparsity - The pixel at the next layer is not connected to all the 100 from the first layer i.e. only a local group is connected to one pixel of next layer. It is not trying to get information from the full image every time. We are harnessing the properties of an image that a group of near-by pixels has better info than grouping distant pixels

So, total parm(definitely size, number, and stride of the kernel will control it)

With a 3x3 kernel,
(3 * 3) + 1 per kernel = 10 per kernel
Even with 200 kernels, it will be only 2K as compared to 10K

Answer 3

In the convolution layer of the convolutional neural network (CNN), each output value depends on a small number of input values, known as the sparsity of connections.

In neural network usage, "dense" connections connect all inputs.

By contrast, a CNN is "sparse" because only the local "patch" of pixels is connected, instead using all pixels as an input.