Have you ever seen a neural network without matrices?
I'm asking, because I'm currently building one for educational purposes.
Matrix multiplication is just a simplified notation for a particular set of addition and multiplication operations. You can absolutely represent a neural network without invoking matrix notation, it'll just be really tedious (and it will run slower).
Start with a single perceptron and build your way up from there.