Issue with Convolutional Layer in Python: Getting All Zeros in Output and Terminating at a Certain Iteration

Question

I'm currently working on implementing a convolutional layer in Python for a natural language processing model. However, I've encountered an issue with the convolutional layer that I can't seem to resolve.

The problem is twofold:

1. Getting all zeros in the output: When I run the forward pass of my convolutional layer, I'm consistently getting all zeros in the output at every position. I've verified this by printing the values during execution. This is unexpected because I initialized the output array correctly.

2. Terminating at a certain iteration: Additionally, the forward pass terminates abruptly at a certain iteration, specifically at the 30th iteration. I'm unable to figure out why the loop is exiting prematurely.

Here's a simplified version of my Conv1DLayer class:

import numpy as np

class Conv1DLayer:

    def __init__(self, num_filters, filter_size):
        self.num_filters = num_filters
        self.filter_size = filter_size
        self.conv_filter = np.random.randn(filter_size, 1)

    def loss(self, pred, target):
        # compute loss function
        return np.mean((pred - target) ** 2)

    def forward(self, inputs):
        self.inputs = inputs
        num_inputs = inputs.shape[1]
        output_length = num_inputs - self.filter_size + 1
        self.output = np.zeros((self.num_filters, output_length))
        # Convolution
        # input dim is basically the size of the vocabulary
        print("input dim: ", inputs.shape)
        print("num inputs: ", num_inputs)
        print("filter dim: ", self.conv_filter.shape)
        print("filter size: ", self.filter_size)
        print("output dim: ", self.output.shape)
        print("output length: ", output_length)
        for i in range(output_length):
            if i+self.filter_size > num_inputs:
                break
            receptive_field = inputs[i:i+self.filter_size, 1].toarray()
            print("receptive field dim: ", receptive_field.shape)
            self.output[:, i] = np.dot(receptive_field.T, self.conv_filter)
            print("output at " + str(i) + str(self.output[:, i]))
            self.output[:, i] = np.maximum(0, self.output[:, i])

        return self.output
    
    def backward(self, grad_outputs, learning_rate):
        grad_input = np.zeros(grad_outputs.shape)
        grad_filter = np.zeros(self.conv_filter.shape)

        for i in range(grad_outputs.shape[0]):
            for j in range(self.num_filters):
               receptive_field = self.inputs[i:i+self.filter_size]
               grad_input[i:i+self.filter_size] += self.conv_filter[:, j] * grad_outputs[i, j]
               grad_filter[:, j] += receptive_field * grad_outputs[i, j]

            # Update the weights
            self.conv_filter -= learning_rate * grad_filter

        return grad_input

I've tried various modifications to the code, including checking the shape of input arrays, adjusting the receptive field, and updating the loop condition, but I haven't been able to resolve the issue.

Here's my output

$ python model.py
input dim:  (32, 2010)
num inputs:  2010
filter dim:  (3, 1)
filter size:  3
output dim:  (10, 2008)
output length:  2008
receptive field dim:  (3, 1)
output at 0[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 1[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 2[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 3[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 4[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 5[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 6[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 7[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 8[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 9[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 10[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 11[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 12[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 13[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 14[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 15[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 16[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 17[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 18[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 19[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 20[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 21[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 22[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 23[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 24[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 25[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 26[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 27[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 28[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 29[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (2, 1)
Traceback (most recent call last):
  File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\model.py", line 89, in <module>
    model.train(X, labels, num_epochs=10, batch_size=32)
  File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\model.py", line 44, in train   
    conv_output = self.conv_layer.forward(inputs)
  File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\convolution.py", line 32, in forward
    self.output[:, i] = np.dot(receptive_field.T, self.conv_filter)
ValueError: shapes (1,2) and (3,1) not aligned: 2 (dim 1) != 3 (dim 0)
(tf) 

Expected behavior:

1. The forward pass should compute the convolution correctly, producing non-zero values in the output.
2. The loop should iterate over all valid positions without terminating prematurely.

My assessment I think the issue is very likely just the way I am using indices but neither chatgpt nor bard have been able to fix it so it might be something deeper.

Any kind of help will be much appreciated. Thanks

Answer 1

For second issue:
Your input dimension is (32,2010).

Here, you are running for loop for (output_length) no. of times, which is 2008. In receptive_field, you are making a ndarray of dimension (3, 1) from the first index (basically slicing 1st column) of inputs (which is of length 32), which is why at 31st iteration, receptive_field has a dimension of (2,1).

So, during dot product, your two matrices are receptive_field.T(1,2) and self.conv_filter(3,1). These matrices cannot be multiplied which is why you get the value error of shapes not being aligned.

Edit:
See this code for forward pass. I have considered only 1 filter in this case. You can change it for more number of filters (stack the convolution outputs for different filters in z-axis). Also, I have considered that you want to perform convolution along x-axis.

In convolution, we slide the filter over input and value of convolution is based on a window around xt according to:

In practice, a ranges from 0 to the filter size.

    import numpy as np

class Conv1DLayer:

    def __init__(self, num_filters, filter_size):
        self.num_filters = num_filters
        self.filter_size = filter_size
        self.conv_filter = np.random.randn(filter_size, 1)
        print(self.conv_filter)

    def loss(self, pred, target):
        # compute loss function
        return np.mean((pred - target) ** 2)

    def forward(self, inputs):
        self.inputs = inputs
        rows = inputs.shape[0]
        columns = inputs.shape[1]
        output_length = columns - self.filter_size + 1
        self.output = np.zeros((rows, output_length))
        # Convolution
        # input dim is basically the size of the vocabulary
        print("input dim: ", inputs.shape)
        print("num inputs: ", rows)
        print("filter dim: ", self.conv_filter.shape)
        print("filter size: ", self.filter_size)
        print("output dim: ", self.output.shape)
        print("output length: ", output_length)
        j=0
        while (j < rows):
            i = 0
            while (i+self.filter_size <= columns):
                receptive_field = inputs[j:j+1, i:i+self.filter_size]
                print("receptive field dim: ", receptive_field.shape)
                self.output[j, i] = np.dot(receptive_field, self.conv_filter)
                print("output at " + str(j)+" " +str(i) + " "+ str(self.output[j, i]))
                self.output[j, i] = np.maximum(0, self.output[j, i])
                i=i+1
            j= j+1

        return self.output

You can go through this article for getting a clear picture of 1d convolution.
Hope this helps!