If I execute the commands
my_reg = LinearRegression()
lin.reg.fit(X,Y)
I train my model. To my understanding training a model is calculating coefficient estimators.
I do not really understand the difference between this and e.g.
scipy.stats.linregress(X,Y)
calculating a 'normal' regression that also gives me the coefficient estimators and all the other statistics connected with it.
Could anyone tell me what is the difference here?
They both solve the exact same objective, which is minimizing the mean squared error. However the second method can answer "how confident it is that slope is not zero, i.e. $Y$ is correlated with $X$?" via p-value.
In detail
Lets denote the data as $(X, Y) = \{(x_n, y_n)|x_n \in \mathbb{R}^D, y_n \in \mathbb{R}\}$. And the regression as $\hat{y} = Ax+B$.
Extra quantities returned by scipy.stats.linregress(X,Y) are: rvalue ($r$), and pvalue ($p$).
In statistics, $r^2$ (known as r-squared) measures the "goodness-of-fit" . That is, as regression $\hat{y}=Ax+B$ gets closer to observation $y$, $r^2$ gets closer to $1$. Since it is a function of $y$ and $\hat{y}$, it can be calculated for the first method too. So no difference here.
However, $p$ is specific to second method. scipy.stats.linregress(X,Y) adds a normality assumption to noise, i.e. assumes $\epsilon \sim N(0, \sigma^2)$ where $$\epsilon = y - \overbrace{Ax+B}^{\hat{y}}$$
On the basis of this assumption, it can answer an additional question: "how confident it is that the slope is not zero?". The first method cannot answer this question.
For example, suppose the estimated slope is $2.1$ for both methods, we still cannot tell whether this slope is significant or $Y$ is actually independent of $X$. Unless we look at the value of $p$. For example, for $p < 0.01$ we are confident (at significance level $0.01$) that $Y$ is correlated with $X$, but for $p > 0.1$ we cannot be confident, i.e. slope $2.1$ could be due to chance and $Y$ might be independent of $X$.
This link gives more details on how p-value is actually calculated in second method.
There is no difference in the conceptual sense - both methods calculate linear regression coefficients. The difference lies in the interface - while through scipy.stats you gain the coefficients directly (and it is up to you to put them into an equation to calculate the predictions), scikit-learn wraps them into a model object so that you can use it in a similar fashion to other ML models such as decision trees, for example. (Actually, you can obtain the regression coefficients from the fitted scikit-learn model using my_reg.coef_.)
as I know, OLS algo is sensible to Outliers & Multicollinearity
Limitations of MLE: it systematically underestimate VaR, as is ML estimator of VaR is biased - this is an instance of overfitting
LS need data of Normal Distribution, for MLE are used Non-parametric (real) data
Compared to K-means in Unsupervesed Algos, MLE can work with overlapping classses, K-means only with lineary separable data
Main difference: MLE is considered to be more general algo than LS, and is used in Generalized Linear Models (like ANOVA, ANCOVA - multivariable regression with categorical features, covariates and confounders - e.g. in Control of confounding) for a variety of other distributions from the exponential family for the residuals - no assumption for residuals to follow a conditionally normal distribution
