The importance of a concept in mathematics depends on the circumstances of its application. Sometimes, its importance relies on the fact that it allows you to carry on with what you are doing.
For example, you usually need column independence (independent variables between predictors) because multiple regression will behave badly with highly correlated variables. Even worst, when some of your columns (or rows) are dependent, your matrix is not invertible. Why? Because matrix inversion A^-1 involves the determinant 1/|A|, which is 0 when columns or rows are linearly dependent.
Eigenvalues is a common occurrence in calculations related to maximization/minimization in machine learning. Let's say you are interested in principal component analysis. A very important idea there is dimensional reduction (you have a dataset with many variables and want to reduce the number of variables without losing too much explanatory power.) One solution is to project your data onto a lower dimensional space (e.g. taking your data with 50 variables and reducing them to 5 variables.) Turns out a good projection to use is one that includes as much variation as possible and maximization of this variation results in the eigenvalue equation S u = λ u.
In other cases, you explicitly include the eigenvalue equation of some quantity of interest because in doing so, you're changing the coordinate system in which you represent the variables. Take the case of a (multivariate) Gaussian distribution in which the argument of the exponent is given by Δ = (x-μ)^T Σ (x-μ). If you consider the eigenvalue equation of Σ, the exponent can be written as Δ = y_1^2 / λ_1 + y_2^2 / λ_2 (in two dimensions) This is the equation of an ellipse only if λ_1 and λ_2 are positive. Therefore, you obtain the following graphical interpretation (Bishop, PRML, p.81):
Positive semi-definite matrices are used as a matter of convenience. They are well-behaved and well-understood. For instance, their eigenvalues are non-negative, and if you remember the previous paragraph, the argument Δ required positive eigenvalues. By now, you can see why some concepts are very popular: You need them for your calculations or they need each other.
I can recommend a couple of books:
- Linear Algebra: A Modern Introduction by David Poole
- Understanding Complex Datasets: Data Mining with Matrix Decompositions by David Skillicorn.
The second recommendation is more specialized and requires a decent understanding of the basics, but it is of great help to understand matrix decompositions.