0
$\begingroup$

I'm currently working on implementing a convolutional layer in Python for a natural language processing model. However, I've encountered an issue with the convolutional layer that I can't seem to resolve.

The problem is twofold:

  1. Getting all zeros in the output: When I run the forward pass of my convolutional layer, I'm consistently getting all zeros in the output at every position. I've verified this by printing the values during execution. This is unexpected because I initialized the output array correctly.

  2. Terminating at a certain iteration: Additionally, the forward pass terminates abruptly at a certain iteration, specifically at the 30th iteration. I'm unable to figure out why the loop is exiting prematurely.

Here's a simplified version of my Conv1DLayer class:

import numpy as np

class Conv1DLayer:

    def __init__(self, num_filters, filter_size):
        self.num_filters = num_filters
        self.filter_size = filter_size
        self.conv_filter = np.random.randn(filter_size, 1)

    def loss(self, pred, target):
        # compute loss function
        return np.mean((pred - target) ** 2)

    def forward(self, inputs):
        self.inputs = inputs
        num_inputs = inputs.shape[1]
        output_length = num_inputs - self.filter_size + 1
        self.output = np.zeros((self.num_filters, output_length))
        # Convolution
        # input dim is basically the size of the vocabulary
        print("input dim: ", inputs.shape)
        print("num inputs: ", num_inputs)
        print("filter dim: ", self.conv_filter.shape)
        print("filter size: ", self.filter_size)
        print("output dim: ", self.output.shape)
        print("output length: ", output_length)
        for i in range(output_length):
            if i+self.filter_size > num_inputs:
                break
            receptive_field = inputs[i:i+self.filter_size, 1].toarray()
            print("receptive field dim: ", receptive_field.shape)
            self.output[:, i] = np.dot(receptive_field.T, self.conv_filter)
            print("output at " + str(i) + str(self.output[:, i]))
            self.output[:, i] = np.maximum(0, self.output[:, i])

        return self.output
    
    def backward(self, grad_outputs, learning_rate):
        grad_input = np.zeros(grad_outputs.shape)
        grad_filter = np.zeros(self.conv_filter.shape)

        for i in range(grad_outputs.shape[0]):
            for j in range(self.num_filters):
               receptive_field = self.inputs[i:i+self.filter_size]
               grad_input[i:i+self.filter_size] += self.conv_filter[:, j] * grad_outputs[i, j]
               grad_filter[:, j] += receptive_field * grad_outputs[i, j]

            # Update the weights
            self.conv_filter -= learning_rate * grad_filter

        return grad_input

I've tried various modifications to the code, including checking the shape of input arrays, adjusting the receptive field, and updating the loop condition, but I haven't been able to resolve the issue.

Here's my output

$ python model.py
input dim:  (32, 2010)
num inputs:  2010
filter dim:  (3, 1)
filter size:  3
output dim:  (10, 2008)
output length:  2008
receptive field dim:  (3, 1)
output at 0[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 1[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 2[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 3[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 4[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 5[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 6[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 7[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 8[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 9[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 10[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 11[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 12[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 13[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 14[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 15[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 16[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 17[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 18[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 19[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 20[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 21[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 22[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 23[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 24[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 25[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 26[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 27[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 28[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (3, 1)
output at 29[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim:  (2, 1)
Traceback (most recent call last):
  File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\model.py", line 89, in <module>
    model.train(X, labels, num_epochs=10, batch_size=32)
  File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\model.py", line 44, in train   
    conv_output = self.conv_layer.forward(inputs)
  File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\convolution.py", line 32, in forward
    self.output[:, i] = np.dot(receptive_field.T, self.conv_filter)
ValueError: shapes (1,2) and (3,1) not aligned: 2 (dim 1) != 3 (dim 0)
(tf) 

Expected behavior:

  1. The forward pass should compute the convolution correctly, producing non-zero values in the output.
  2. The loop should iterate over all valid positions without terminating prematurely.

My assessment I think the issue is very likely just the way I am using indices but neither chatgpt nor bard have been able to fix it so it might be something deeper.

Any kind of help will be much appreciated. Thanks

$\endgroup$

1 Answer 1

0
$\begingroup$

For second issue:
Your input dimension is (32,2010). enter image description here

Here, you are running for loop for (output_length) no. of times, which is 2008. In receptive_field, you are making a ndarray of dimension (3, 1) from the first index (basically slicing 1st column) of inputs (which is of length 32), which is why at 31st iteration, receptive_field has a dimension of (2,1).

enter image description here

So, during dot product, your two matrices are receptive_field.T(1,2) and self.conv_filter(3,1). These matrices cannot be multiplied which is why you get the value error of shapes not being aligned.

Edit:
See this code for forward pass. I have considered only 1 filter in this case. You can change it for more number of filters (stack the convolution outputs for different filters in z-axis). Also, I have considered that you want to perform convolution along x-axis.

In convolution, we slide the filter over input and value of convolution is based on a window around xt according to:
enter image description here
In practice, a ranges from 0 to the filter size.

    import numpy as np

class Conv1DLayer:

    def __init__(self, num_filters, filter_size):
        self.num_filters = num_filters
        self.filter_size = filter_size
        self.conv_filter = np.random.randn(filter_size, 1)
        print(self.conv_filter)

    def loss(self, pred, target):
        # compute loss function
        return np.mean((pred - target) ** 2)

    def forward(self, inputs):
        self.inputs = inputs
        rows = inputs.shape[0]
        columns = inputs.shape[1]
        output_length = columns - self.filter_size + 1
        self.output = np.zeros((rows, output_length))
        # Convolution
        # input dim is basically the size of the vocabulary
        print("input dim: ", inputs.shape)
        print("num inputs: ", rows)
        print("filter dim: ", self.conv_filter.shape)
        print("filter size: ", self.filter_size)
        print("output dim: ", self.output.shape)
        print("output length: ", output_length)
        j=0
        while (j < rows):
            i = 0
            while (i+self.filter_size <= columns):
                receptive_field = inputs[j:j+1, i:i+self.filter_size]
                print("receptive field dim: ", receptive_field.shape)
                self.output[j, i] = np.dot(receptive_field, self.conv_filter)
                print("output at " + str(j)+" " +str(i) + " "+ str(self.output[j, i]))
                self.output[j, i] = np.maximum(0, self.output[j, i])
                i=i+1
            j= j+1

        return self.output


You can go through this article for getting a clear picture of 1d convolution.
Hope this helps!

$\endgroup$
2
  • $\begingroup$ First of all thank you so much for the detailed reply. For the second issue, this is the first time I have every tried implementing a Conv layer and all I had was an idea. I thought creating a 1d array of random values of the size filter-size should be the way and then I would keep on multiplying this to the receptive field at each position in the input matrix. But that doesn't seem to work since at one point there's a column lacking. I am sure my implementation of the filter itself is wrong. Can you point me in the correct direction here? Thanks. $\endgroup$ Jul 15, 2023 at 12:53
  • $\begingroup$ I have added a sample code and an article link. Hope it helps. $\endgroup$
    – shivani
    Jul 15, 2023 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.