Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Looking at the LDA of the Iris Dataset, which looks like this:
It's understandable that the 3 types of flowers have succesfuly been seperated into categories, with versicolor and virginica slightly overlapping. My question is, how does this help us in a practical way? What can we do with the information, now that we've seperated the 3 categories like this. It's great to know that Setosa is apparent for values between 5 and 10, but what do those values mean?
You could be asking yourself the same questions using PCA! Usually these linear transformations (or even non-linear for that matter, PCA, LDA, ICA, UMAP etc.) are used to reduce the high dimensional features space to serve one or multiple purposes. For example in PCA one could use it to compress the high-dimensional feature space to lower (usually 2 or 3) by finding the directions of maximal variance, either for better separation or a simple compression or better explainability via projected visualization for human eyes. Sometimes they are served for better class separation like the case of LDA, in a supervised fashion (in contrast to PCA). In case you missed it check out Sebastian Raschka's post on the difference between LDA and PCA for dimensionality reduction (be careful with LDA assumptions).
In order to answer the question about the axes, we better touch briefly on what LDA is aiming to do, borrowing the explanation from this blog post:
LDA aims to find a new feature space to project the data in order to maximize classes separability. The first step is to find a way to measure this capacity of separation of each new feature space candidate. In 1988, a statistician called Ronald Fisher proposed the following solution: Maximize the function that represents the difference between the means, normalized by a measure of the within-class variability. The Fisher’s propose is basically to maximize the distance between the mean of each class and minimize the spreading within the class itself. Thus, we come up with two measures: the within-class and the between-class.
[The Answer]: As in the case of PCA: the first axis (by convention PCA 1) represents the axis holding the maximum variance in the data, PCA 2 the second in the rank and so forth. The same analogy holds true to LDA. Here though, LDA 1 (your x-axis in the graph, again by convention and the visual judgement) accounts for the maximum variation among the classes, and LDA 2 the second and so forth (still confused, you might one to check this video). You can define a simple linear model to separate them, that is it, that perhaps you couldn't using the original features.
[Bonus]: Although I am finished with the answer, and the following it is not super relevant, but I would like to bring a super nice example what these techniques might be able to answer from data. In the case of, again PCA, Tableau team is showing such capabilities, see this video. In real world problems, not a simple IRIS, these will come handy, and surely these possible even outside Tableau, but requires more efforts.
Intrinsically the values don't mean anything specific: it is the result of the axis combination. And most of the time, the components hold more information than the projection of the points on the new axis system.
We can easily think of an example where LDA components are informative. Imagine you have multiple features for human bodies (weight, height, length of arms, etc) and you want to separate the obese people (1) from not obese people (0). Weight is not very helpful to separate people, nor is Height on.
But LDA would probably give you a first axis having a strong composition of weight and height. What does that mean? That combination of height and weight gives you a better way to determine if a person is obese or not.
It could be informative for you, as an analyst or for a classification algorithm later on.
