Great answer by Leevo, just let me point out one thing: Perfect multicollinearity means that one variable is a linear combination of another. Say you have $x_1$ and $x_2$, where $x_2 = \gamma x_1$. This causes various problems as discussed in this post.
The main takeway (to put it simple) is, that $x_1$ and $x_2$ basically carry the same information (just "scaled" by $\gamma$ in the case of $x_1$). So there is no benefit of including both. In fact there is a problem with including both since multicollinearity will "confuse" the model because there is no unique effect of $x_1, x_2$, when considered jointly, on some outcome $y$.
Look the following situation (R code):
y = c(5,2,9,10)
x1 = c(2,4,6,8) ### = 2 * x2
x2 = c(1,2,3,4) ### = 0.5 * x1
cor(x1, x2, method = c("pearson"))
The correlation between $x_1$ and $x_2$ equals 1 (so of course a linear combination). Now when I try to make a simple linear OLS regression:
lm(y~x1+x2)
The result is:
Coefficients:
(Intercept) x1 x2
1.0 1.1 NA
The second term has been dropped by R (due to perfect multicollinearity).
We can run a regression on each term separately:
Call:
lm(formula = y ~ x1)
Coefficients:
(Intercept) x1
1.0 1.1
...and...
Call:
lm(formula = y ~ x2)
Coefficients:
(Intercept) x2
1.0 2.2
Now you can see that the coefficient for $\beta_2$ is simply $2\beta_1$ because $x_1$ is $2 x_2$. So nothing to learn from including both, $x_1, x_2$ since there is no additional information.
Basically the same problem can occur if correlation between $x_1,x_2$ is really high. See some more discussion in this post. Thus given strong correlation, one should be cautious to include both variables. The reason is that in this case, your model cannot really tell apart the effect of $x_1$ and $x_2$ on some outcome $y$, so that you may end up with weak predictions (among other problems).
