3
$\begingroup$

Let's say that I have sparse feature vectors and I'd like to use dimensionality reduction in order to visualize them more easily.

Dimensionality reduction techniques like PCA will estimate colinearity between the features given the data. What if I have some prior knowledge on the colinearity between my features? As in, I would be able to create an approximate distance matrix between my features, and therefore between my data points.

I am aware that if features are actually colinear, methods like PCA will find a way to reduce them, however, I am afraid that I don't have enough data points to infer that colinearity strictly from the data and that PCA would remove meaningful features and keep uninformative ones.

Let's say that my data looks like something like this:

$x_0 = \{1,NaN,-1,NaN\}$

$x_1 = \{1,NaN,NaN,-1\}$

In my case, I know that the 3rd and 4th features are colinear and that the distance between $x_0$ and $x_1$ is close to zero. Therefore, they should be mapped to a very similar data point in a lower-dimensional space.

Can PCA still do it with so few data points? Are there ways to "force" the known colinearity or distance measures? T-SNE perhaps?

Improve this question
$\endgroup$
3
  • $\begingroup$ If you know that 2 features are colinear why would you want to use both of them? Just slows down your training procedure and you won't get more data. $\endgroup$
    – AR_
    Commented Oct 23, 2017 at 8:47
  • 1
    $\begingroup$ That's exactly my issue. Let's say that I have two sentences that use completely different words but all these words are synonymous. This means that my features (here bag of words) are completely different, but because the words are synonymous, I would still like to have both vectors be mapped to the same point in a lower-dimensional space. I am looking for ways to "force" this knowledge of colinearity into a dimensionality-reduction technique $\endgroup$
    – Valentin Calomme
    Commented Oct 23, 2017 at 8:55
  • 1
    $\begingroup$ @AR_ : sometimes the colinearity isn't strict, sometimes the relation ship isn't known exactly. Sometimes you have way too much variables to deal with them all manually. May be the question is a bit too narowly formulated, but the problem is of tremendous importance for tabular data. $\endgroup$
    – Lucas Morin
    Commented Dec 6, 2019 at 11:29

1 Answer 1

Reset to default
1
+50
$\begingroup$

One option is to use Multidimensional scaling (MDS) for dimensionality reduction. MDS can create a visualization of the relative positions for data based on the distance between the data points.

In your example, the data points that have no distance between them in several dimensions will be projected close to each other.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.