0
$\begingroup$

Ok, maybe this is a really dumb question, but has anyone ever considered extending some of the statistical learning methods to live in some space other that $\mathbb R^n$? My guess is that you're just shuffling parameters around, but I wonder if there's any intuition to be gained.

This question came up as I was working through some course notes on neural nets, and realized that computer scientists may be even more cavalier with tensor indices than physicists :)

$\endgroup$
2
  • $\begingroup$ Look into manifold learning, information geometry, natural gradient methods, tensor methods, group theoretical methods, complex valued neural networks, etc. It's all there if you look. $\endgroup$ – Emre May 30 '17 at 7:12
  • $\begingroup$ Thanks for the refs @Emre, just learning the details of neural networks. $\endgroup$ – BenDundee May 30 '17 at 13:52
2
$\begingroup$

Essentially, each non-linear layer in a neural network is a map from $\mathbb{R}^n$ input to a $\mathbb{R}^m$ output. There is no requirement for these dimensions to be separate and uncorrelated, nor for the map to be reversible.

The best performance is usually found if each layer is roughly scaled to approximate a normal distribution with mean 0 and standard deviation 1. However, that is an independent issue to any kind of space-like mapping.

If you know a suitable mapping that relates to your problem and would simplify its expression, then you can apply a spacial transform between co-ordinate systems (typically by mapping the input features) and use it within the neural network. You could map co-ordinates to any curved space if you think it might help. Choosing between polar or cartesian co-ordinates for features is one choice that may crop up for example.

However, even without making this helpful mapping, you can analyse a neural network layer-by-layer, and see that it can effectively learn these kind of spacial mappings from the training data anyway. This is an intuition that has been called the "manifold hypothesis" - the link has some images showing how a neural network might learn to separate classes depending on how they are arranged in feature space, and this article from a DARPA researcher has a nice series of images showing successive transformations of a 2D spiral that allow the last layer to perform linear separation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.