How to reduce position changes after dimensionality reduction?

Question

Disclaimer: I'm a machine learning beginner.

I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing their positions after the modification and updating the 2D-plot. The logic already works, but each iterative visualization is very different from the previous one even though only 1 out of 28.000 features in 1 data point changed.

Some details about the project:

• ~1000 text documents/data points
• ~28.000 tfidf vector features each
• must compute pretty quickly (let's say < 3s) due to its interactive nature

Here are 2 images to illustrate the problem:

Step 1:

Step 2:

I have tried several dimensionality reduction algorithms including MDS, PCA, tsne, UMAP, LSI and Autoencoder. The best results regarding computing time and visual representation I got with UMAP, so I sticked with it for the most part.

Skimming some research papers I found this one with a similar problem (small change in high dimension resulting in big change in 2D): https://ieeexplore.ieee.org/document/7539329 In summary, they use t-sne to initialize each iterative step with the result of the first step.

First: How would I go about achieving this in actual code? Is this related to tsne's random_state?

Second: Is it possible to apply that strategy to other algorithms like UMAP? tsne takes way longer and wouldn't really fit into the interactive use case.

Or is there some better solution I haven't thought of for this problem?

Answer 1

You can initialize a UMAP embedding with a custom set of initial positions, so potentially you can initialise step 2 with the embedding from step 1 (with random positions for the new points).

Answer 2

To reduce position change, it is crucial to know how t-SNE works.

t-SNE is a projection from a high-dimensional space to a lower one, generally 2D or 3D.

For simplification purposes, let's take a 2D low-dimensional space.

This 2D low-dimensional space evolves with every iteration until finding a balance that represents the high-dimensional space as well as possible.

The low-dimensional space is a space of probability with just a relative distance meaning between points.

Consequently, if you fix some key points like the gravity centers of each cluster from the first result, and reuse their position in the new t-SNE generations' 2D low-dimensional map, the new data should organize themselves according to those fixed points.

All you have to do is to take every new interaction and modify the position of the key points to keep them fixed. The other points should progressively move according to those key points, and you should get at the end comparable results.