I'm currently working on implementing a convolutional layer in Python for a natural language processing model. However, I've encountered an issue with the convolutional layer that I can't seem to resolve.
**The problem is twofold:**
1. Getting all zeros in the output: When I run the forward pass of my convolutional layer, I'm consistently getting all zeros in the output at every position. I've verified this by printing the values during execution. This is unexpected because I initialized the output array correctly.
2. Terminating at a certain iteration: Additionally, the forward pass terminates abruptly at a certain iteration, specifically at the 30th iteration. I'm unable to figure out why the loop is exiting prematurely.
Here's a simplified version of my Conv1DLayer class:
```
import numpy as np
class Conv1DLayer:
def __init__(self, num_filters, filter_size):
self.num_filters = num_filters
self.filter_size = filter_size
self.conv_filter = np.random.randn(filter_size, 1)
def loss(self, pred, target):
# compute loss function
return np.mean((pred - target) ** 2)
def forward(self, inputs):
self.inputs = inputs
num_inputs = inputs.shape[1]
output_length = num_inputs - self.filter_size + 1
self.output = np.zeros((self.num_filters, output_length))
# Convolution
# input dim is basically the size of the vocabulary
print("input dim: ", inputs.shape)
print("num inputs: ", num_inputs)
print("filter dim: ", self.conv_filter.shape)
print("filter size: ", self.filter_size)
print("output dim: ", self.output.shape)
print("output length: ", output_length)
for i in range(output_length):
if i+self.filter_size > num_inputs:
break
receptive_field = inputs[i:i+self.filter_size, 1].toarray()
print("receptive field dim: ", receptive_field.shape)
self.output[:, i] = np.dot(receptive_field.T, self.conv_filter)
print("output at " + str(i) + str(self.output[:, i]))
self.output[:, i] = np.maximum(0, self.output[:, i])
return self.output
def backward(self, grad_outputs, learning_rate):
grad_input = np.zeros(grad_outputs.shape)
grad_filter = np.zeros(self.conv_filter.shape)
for i in range(grad_outputs.shape[0]):
for j in range(self.num_filters):
receptive_field = self.inputs[i:i+self.filter_size]
grad_input[i:i+self.filter_size] += self.conv_filter[:, j] * grad_outputs[i, j]
grad_filter[:, j] += receptive_field * grad_outputs[i, j]
# Update the weights
self.conv_filter -= learning_rate * grad_filter
return grad_input
```
I've tried various modifications to the code, including checking the shape of input arrays, adjusting the receptive field, and updating the loop condition, but I haven't been able to resolve the issue.
**Here's my output**
```
$ python model.py
input dim: (32, 2010)
num inputs: 2010
filter dim: (3, 1)
filter size: 3
output dim: (10, 2008)
output length: 2008
receptive field dim: (3, 1)
output at 0[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 1[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 2[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 3[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 4[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 5[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 6[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 7[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 8[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 9[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 10[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 11[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 12[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 13[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 14[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 15[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 16[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 17[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 18[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 19[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 20[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 21[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 22[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 23[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 24[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 25[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 26[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 27[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 28[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (3, 1)
output at 29[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
receptive field dim: (2, 1)
Traceback (most recent call last):
File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\model.py", line 89, in <module>
model.train(X, labels, num_epochs=10, batch_size=32)
File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\model.py", line 44, in train
conv_output = self.conv_layer.forward(inputs)
File "C:\Users\maste_0c98yk4\OneDrive\Desktop\Projects\Natural Language Processing Model - Sentiment Analysis\convolution.py", line 32, in forward
self.output[:, i] = np.dot(receptive_field.T, self.conv_filter)
ValueError: shapes (1,2) and (3,1) not aligned: 2 (dim 1) != 3 (dim 0)
(tf)
```
**Expected behavior:**
1. The forward pass should compute the convolution correctly, producing non-zero values in the output.
2. The loop should iterate over all valid positions without terminating prematurely.
**My assessment**
I think the issue is very likely just the way I am using indices but neither chatgpt nor bard have been able to fix it so it might be something deeper.
Any kind of help will be much appreciated. Thanks