# Training Neural Networks with unknown length of input

I'm currently going into the world of machine learning and Neural Networks, thanks to synaptic (js) that interests me a lot.

So I read a lot, wikipedia links and synaptic's NN 101, but there's a lot of basics questions that I don't understand (but I'd like to) in the use of machine learning (NN) and the point of these technologies.

Let's say, I wan't my network to (kind of) learn (something like) gravity, so to train it I set in input 10 objects with a mass, and a position x, y (and z) and I set output the new x, y (and z) of each objects. I guess I should give it several configurations and everything but here is the question; can it, then, be able to compute the interactions between 10000, 100000 objects?

At this stage in my learning, what I don't clearly get is what is the point of teaching/training neurons to compute XOR like it's shown in synaptic's documentation:

var trainingSet = [
{
input: [0,0],
output: [0]
},
{
input: [0,1],
output: [1]
},
{
input: [1,0],
output: [1]
},
{
input: [1,1],
output: [0]
},
];

var trainer = new Trainer(myNetwork);
trainer.train(trainingSet);


Were we just give it all the possible inputs and outputs to a XOR.

Well, as I'm all new to the technologies I think my questions are full of non-sense and everything but thanks for reading and help you might bring :)

Gravity

Basically you'd like to find a function that maps an input (an object with a mass and a location) to an output (new location). It's not necessary to have one set of input neurons for each different object. It's sufficient to encode the input variables generically for all objects.

Like a function:

$$f( x, y, z, mass ) \to ( x_n, y_n, z_n )$$

XOR

Training the XOR function is special as the early neural networks, the perceptrons, were unable to learn the XOR function (because it is not linear separable). Multi-Layer Perceptrons however are able to learn the XOR function. This is just to demonstrate that the NN implementation of Synaptic is capable of learning problems that are not linear separable

• Ok thanks, that's somehow more clear, especially for the XOR. For the gravity thing, I don't know if it's a good example, I guess machine learning doesn't really apply as we know how gravity works we don't really need machine learning, right? The question was more about teaching something with a small array of inputs, and apply it to a larger array. – Cohars Dec 10 '14 at 16:26
• There are different strategies for neural networks to work with unknown data, and there are also different strategies to let a neural network grow. But all of them wouldn't really fit to the question. – Regenschein Dec 13 '14 at 9:00

This is the fundamental challenge to all data modeling. We don't just want to memorize the the link between a given input and a given output (otherwise you wouldn't be modeling data, you'd be memorizing 1:1 connections with a dict / hash / relational database table / etc). We want to capture the underlying pattern in the data from only looking at the training data.

Let's expand a little on your gravity example. You have your 10 training samples showing the start and ending position of an object dropped. For consistency, let's say the object was dropped the moment the object's location was initially recorded and the ending location was recorded at some precise time interval later (but before the object hit the ground). Let's also say the model (neural network in this case) managed to precisely learn the expected change in location since it just comes down to subtraction in one axis. You can show it another 10, 100, 1000 examples that all leverage the connection found and your model will keep performing well.

Why not keep going to 10k, 100k, or even more samples? Theoretically, if you managed to isolate the connection and run the experiment the same way each time, your model will always work. But realistically, something is going to eventually change in the system. You hire a new lab assistant who tends to press the 'record location' button well after having dropped the object (giving the object more initial velocity, which you won't notice having only recorded location). Maybe you lost your initial ball and had to use something else which is lighter and catches the wind more (so it goes slower). .... the longer you run the experiment, the more small changes will creep into your system. Eventually these changes will alter the connection enough to make your initial model wrong.

When modeling data, we want to capture the underlying patterns and acknowledge that the model only matters as long as those underlying patterns stay relevant. It's not really about the number of samples. It's about the connections / the model itself. The number of samples just happens to be one of the better proxies we have - the more samples you use, the more confident you have some underlying pattern. 'Statistical validity' is one stab at solving this, though it's validity is still up for question in the era of big data. There is plenty of work done trying to solve for how to gain confidence in good generalization in neural networks specifically, but it's still very much an open question.

For a different example, if you're looking at user behavior, you'll see differences between day and night; weekdays and weekends; summer and winter; year of a person's life; culture a person grew up in... even if you prove you found a pattern in your initial sample, the system will eventually change and it's up to luck whether the connection(s) you found are a part of the system that changed or a part of the system that didn't.