The dimensions of the low dimensional space have no meaning. Imagine rotating the whole low-dimensional map, the pair-wise distances do not change and consequently the pair-wise probabilities remain equal. Likewise, rotating the high-dimensional map does not change their pair-wise probabilities. Hence, rotating either does not change the quality of the solution. 

This illustrates that the t-SNE loss function is soley based on the distances between points. Furthermore, it only defines a mapping from a specific set of high-dimensional points to a set of low dimensional points. That's the core of the answer. t-SNE does not define a function that defines a relation between the high-dimensional space (instead of points) and the low-dimensional space, it only defines a mapping between the points. It finds this mapping by looking at the (gaussian) distances between points in the high-dimensional map and the (t-Student) distances in the low-dimensional map.

Also: [the paper][1]. 


  [1]: https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf