# Dimension reduction techniques in R that do not use the full distance matrix

I try to apply non-linear dimension reduction in R. As usual in machine learning I have a large data set (100 K rows). I tried the packages RDRToolbox and vegan:

library( RDRToolbox )
res.lle = LLE(as.matrix(temp), dim = 3 , k = 10)
res.iso = Isomap(as.matrix(temp), dim = 3 , k = 10)

library( vegan )
isomap(dist(temp), ndim=10)


The first algoritms use the distance matrix internally and crash as they want to allocate 32GB. The second approach needs the distance matrix and crashes too.

Are there any tricks to do non-linear dimension reduction either with these algorithms? Or are there packages that do not use the raw and full distance matrix? Such approaches have to crash in my setting on my machine.

The only algorithm that works for me is Rtsne that appearantly does not use the whole distance matrix. largeVis was too slow so far to let it finish.

You can try using an auto-encoder which also is a non-linear dimension reduction technique. It uses a neural network framework to find the most efficient transformation from $p$ dimensions down to whatever you choose. It then finds how well it can reconstruct the origianl $p$ variables, and keeps tuning until it can optimally recreate the original values as "good" (lowest MSE) as possible.