2
$\begingroup$

in Spark, there is a RowMatrix.columnSimilarities() method (see http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/linalg/distributed/RowMatrix.html#columnSimilarities()) that returns "An n x n sparse upper-triangular matrix of cosine similarities between columns of this matrix".

How should I read it? If I try to implement an example from https://stackoverflow.com/a/1750187 as following:

JavaRDD<Vector> rows = sc.parallelize(Arrays.asList(
    new DenseVector(new double[]{2, 1, 0, 2, 0, 1, 1, 1}),
    new DenseVector(new double[]{2, 1, 1, 1, 1, 0, 1, 1})
));

RowMatrix mat = new RowMatrix(rows.rdd());  
List<Vector> sims = mat.columnSimilarities().toRowMatrix().rows().toJavaRDD().collect();
for(Vector v: sims) {
    System.out.println(v);
}

I get this

(8,[6,7],[0.7071067811865475,0.7071067811865475])
(8,[1,2,3,4,5,6,7],[0.9999999999999998,0.7071067811865475,0.9486832980505137,0.7071067811865475,0.7071067811865475,0.9999999999999998,0.9999999999999998])
(8,[2,3,4,5,6,7],[0.7071067811865475,0.9486832980505137,0.7071067811865475,0.7071067811865475,0.9999999999999998,0.9999999999999998])
(8,[7],[0.9999999999999998])
(8,[4,5,6,7],[0.4472135954999579,0.8944271909999159,0.9486832980505137,0.9486832980505137])
(8,[6,7],[0.7071067811865475,0.7071067811865475])
(8,[3,4,6,7],[0.4472135954999579,1.0,0.7071067811865475,0.7071067811865475])

How should I interpret it? How do I get the cosine angle 0.822 from this, as mentioned in the referenced StackOverflow post?

Thanks!

$\endgroup$
2
$\begingroup$

Solution is to transform the matrix:

JavaRDD<Vector> rows = jsc.parallelize(Arrays.asList(
    new DenseVector(new double[]{2, 2}),
    new DenseVector(new double[]{0, 1}),
    new DenseVector(new double[]{1, 1}),
    new DenseVector(new double[]{1, 0}),
    new DenseVector(new double[]{0, 1}),
    new DenseVector(new double[]{2, 1}),
    new DenseVector(new double[]{1, 1}),
    new DenseVector(new double[]{1, 1})
));
​
RowMatrix mat = new RowMatrix(rows.rdd());
List<Vector> sims = mat.columnSimilarities().toRowMatrix().rows().toJavaRDD().collect();
for(Vector v: sims) {
  System.out.println(v); //(2,[1],[0.8215838362577492])
}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.