So I have looked at some of the literature on neural networks and read some chapters, but the learning curve is so steep that I have had trouble even getting started on designing the neural network to solve my problem.

From what I understand, the architecture (or arrangement of neurons and their connections) should be designed according to the nature of the problem to be solved. Other parameters to be set according to the nature of the problem include the loss index (how the error will be calculated and if there should be a regularization term), whether or not there should be any scaling/unscaling, bounding, or conditions, and the training algorithm (such as the quasi-Newton method).

The particular type of problem I am interested in is using neural networks to figure out unknown functions (with unknown complexity) that input and output integers (as opposed to continuous values), given a large collection of inputs and outputs.

An example function takes 4 byte inputs and returns 2 byte outputs. This is done by first taking the first 2 bytes of input and XORing them with the last 2 bytes, to produce an intermediate result. This two byte value is then XORed with a copy of itself that is shifted left by 5 bits. This result is then XORed with a copy of itself shifted right by 7 bits. Then this result is then XORed with a copy of itself shifted left by 2 bits, and this value is outputted by the function, giving the final 2-byte result. Note: More than one unique input can produce the same output.

So given a large set of inputs and outputs of an unknown function, the neural network should then optimize itself to reproduce this function given new inputs. I am not sure how to get started designing this neural network, and I am not sure fully reading through neural networks textbooks is the optimal way to get started. I am using a software library meant for designing neural networks, and I can simply set the network architecture and the parameters described above. How much theory do I need to know in order to get started solving my problem? Where should I start with learning how to design this neural network?

EDIT: The main goal of all this is to use an existing tool to simplify producing an output (a 2-byte code, in the example above) given a new input, in a situation where the function is unknown. Neural networks seem to match the function-finding-through-trial-and-error characteristics that I need. This tool should be able to try all sorts of possible functions of increasing complexities in order to mimic the working of the actual unknown function.

  • 1
    $\begingroup$ Your example function seems to have some things in common with cryptographic compression primitive, or maybe a PRNG. Neural networks (and statistics-based models that learn a function from examples, which is the majority of ML algorithms) would generally be a bad fit to learn/predict such functions. You may be better taking a step back and explain more about your discrete functions and why you want to predict them (i.e. what is your goal? To learn ML, or specifically model a type of function), then ask about a suitable approach. $\endgroup$ – Neil Slater Jun 16 '18 at 17:56
  • $\begingroup$ @NeilSlater Yes, I got most of the example from the xorshift PRNG, and the actual problem I am trying to solve is to recreate an unknown, not-very-sophisticated (since there are many examples of collisions in my set of known input-output pairs) cryptographic compression function. To do this manually is possible but it would be a very long and tedious process. I recognize that my problem is "ill-posed", in that small changes input produce large changes in output, but could an increasing number of hidden layers, in the case of the feed-forward architecture, be able to handle this? $\endgroup$ – user59513 Jun 16 '18 at 18:49
  • 1
    $\begingroup$ Using a NN (or any other statistical learner) to create or analyse a cryptographic function directly is probably going to go nowhere. Doubly so if you are hoping to skip the theory understanding steps and hope to use whatever a NN library produces you from source data directly. I'm not a crypto expert, but I expect that automating modern cryptanalysis would likely only use a small amount of direct ML/stats, to e.g. show that a routine does not have certain weaknesses. Historic ciphers, such as substitutions and Vigenère, were more tractable using stats. $\endgroup$ – Neil Slater Jun 16 '18 at 21:07
  • $\begingroup$ Cross-posted: datascience.stackexchange.com/q/33255/8560, stats.stackexchange.com/q/351687/2921. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. $\endgroup$ – D.W. Jun 18 '18 at 6:31

Here is a good way to get going:

  1. C++ or c#
  2. spend a week learning vectors and matrices.
  3. Be able to code a "class vector" and "class matrix"
  4. Get intuition on how matrices represent weights between 2 layers
  5. Learn the chain rule and the "tree diagram" when computing composite (nested) functions
  6. Spend a week learning backprop. Be able to compute derivatives for all layers, using pen an paper a toy example with 4 layers, 3 neurons each.
  7. be able to make code that reads a .txt file character by character.
  8. Learn about one-hot, to describe your character to the network (it only understands vectors) - or see how your byte-problem might be used instead of characters.
  9. Learn about hard-max softmax and their derivatives, tanh and sigmoid
  10. code fwd prop
  11. code bkprop
  12. test if it can predict the next char
  13. debug
  14. debug
  15. DEBUG
  16. Gradient checking
| improve this answer | |
  • 1
    $\begingroup$ These are good drills! $\endgroup$ – Elliot Jun 17 '18 at 16:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.