1
$\begingroup$

What is meant by energy spectrum in LSI(Latent Semantic Indexing)?

I am doing topic modeling with gensim's LsiModel, and part of the output per chunk is the following:

INFO : preparing a new chunk of documents
INFO : using 100 extra samples and 2 power iterations
INFO : 1st phase: constructing (100000, 600) action matrix
INFO : orthonormalizing (100000, 600) action matrix
INFO : 2nd phase: running dense svd on (600, 20000) matrix
INFO : computing the final decomposition
INFO : keeping 500 factors (discarding 6.560% of energy spectrum)
INFO : merging projections: (100000, 500) + (100000, 500)
INFO : keeping 500 factors (discarding 0.843% of energy spectrum)
INFO : processed documents up to #1400000

The above output is from 1,400,000 documents into the process, (out of aproxx. 3,500,000), and it appears to discard less and less for each chunk. The 2nd chunk was higher:

INFO : keeping 500 factors (discarding 6.556% of energy spectrum)
INFO : merging projections: (100000, 500) + (100000, 500)
INFO : keeping 500 factors (discarding 13.469% of energy spectrum)
INFO : processed documents up to #40000

I am not sure whether the discarding X % of energy spectrum is better with high or low numbers. Is the "energy" analogous to entropy? Does discarding mean it looses information, or does it mean the Singular Value Decomposition is getting better and better with more information? (Or none of the above)

Improve this question
$\endgroup$
4
  • $\begingroup$ It's the squared sum of eigenvalues. Discarding does mean losing information. Losing energy in an approximation is mostly bad, but judicious application can eliminate noise and improve generalization. en.wikipedia.org/wiki/Spectrum_of_a_matrix en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm $\endgroup$
    – Emre
    Jun 13, 2017 at 18:46
  • $\begingroup$ Thanks for your helpful comment, @Emre. I haven't found the term "energy" related to eigenvalues, LSI or SVD in any intrinsic context. Thanks again. $\endgroup$
    – Thomas Fauskanger
    Jun 14, 2017 at 7:32
  • $\begingroup$ It's engineering terminology. If you search the code, you'll see all invocations of clip_spectrum invoke the squared singular values (I said eigenvalues, which applies to square matrices). Welcome to the site, and good luck. $\endgroup$
    – Emre
    Jun 14, 2017 at 7:40
  • $\begingroup$ Thanks, I think I understand the term usage. It literally clips the end of the list of singular values/eigenvalues, and because they are sorted in decreasing order, the loss is the ratio of the clipped singular values, i.e. their contribution to the sum of the spectrum of the matrix. (And singular values are the square root of eigenvalues when that makes sense.) $\endgroup$
    – Thomas Fauskanger
    Jun 14, 2017 at 7:57

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.