0
$\begingroup$

I want to implement a Randomised Neural Network (alt. Neural Network with Random Weights (NNRW)) in keras based on the following paper: https://arxiv.org/pdf/2104.13669.pdf

Essentially the idea is the hidden layers are fixed randomly and only the output layer is optimized. (leading to a simple Least Squares solution).

I am familiar with using the Sequential API in keras to create models although I'm not sure how I would go about 'fixing' the hidden layers and only focus on optimising the output layer.

My last ditch attempt would be to simply code the network manually in NumPy, but I wanted to ask if anyone had any ideas on doing so in Tensorflow

$\endgroup$

1 Answer 1

1
$\begingroup$

if you're new to deep learning frameworks, firstly I'd recommend pytorch. Although Tensorflow has more capabilities, most of them aren't required for beginners, and with growing experience and sophistication of your project, you still may switch to tf later, in case you need to really dive deep. Once you know pytorch, it will be a lot easier to get into tf.

Now, about your model: If you're planning to implement a simple feedforward network, you can easily use the nn.Sequential module in pytorch to do so. Each layer in your sequential stack of layers has parameters (in most cases called weight and bias). These are of type torch.Parameter, which essentially means that the optimizer in training will recognize that they are model parameters that should be upgraded with gradient descent. Whether a parameter will be considered in the update step is determined by the attribute requires_grad = True / False. What you essentially want to do is to turn off upgrading of parameters (which is typically called "freezing") that belong to the hidden layers. This is plain easy. You simply loop through your layers with model.named_parameters() and set requires_grad=False for the ones that you want to keep fixed / random. Make sure you initialize your model parameters randomly (which is done by default, but there are different techniques to initialize them, which may alter your outcome).

If you have absolutely no idea what I am talking about, simply do the 60minute intro tutorial in pytorch and you will be good to go. Knowing python is a prerequisite of course ;)

For more sophisticated randomized models, you might be interested in looking into echo state networks, in case you need something like a recurrent network.

Good luck!

$\endgroup$
4
  • $\begingroup$ Thank you for your reply! I had heard of PyTorch but was not aware of this useful feature! I'll be sure to read up on the documentation :D I'm actually also researching ESNs as part of the project I'm working on :) $\endgroup$
    – SwagCakes
    Commented Apr 6, 2022 at 16:21
  • $\begingroup$ the feature is definitely also available in tf, but i'd still recommend pytorch for the start. :) btw, for ESNs there is even a pytorch library on github called echotorch i think. haven't tried it, so you should check it before, but it should be easy to use and handles all the initializations of the weights with the constraints on the weight matrices and so on. $\endgroup$
    – user101893
    Commented Apr 6, 2022 at 16:35
  • $\begingroup$ Ah perfect, I'll take a looksie! I was actually planning on playing around with the effect different weight initialisations have on the overall performance of the network, I presume that's something I'd be able pass in as an argument on echotorch? $\endgroup$
    – SwagCakes
    Commented Apr 7, 2022 at 19:05
  • $\begingroup$ hey, as said, i havent tried echotorch, but youll figure it out. it wont be difficult to change the initialization. take a look into pytorch's initialization mechanisms in the docs as well. there are plenty. $\endgroup$
    – user101893
    Commented Apr 8, 2022 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.