The function you have described is a loss function. It is the function which we want to minimize in order to train our model. The loss function is called ordinary least squares. We can also see that the model that we are trying to fit is a linear function with model parameters $\beta$. This is an optimization function and regularizarion is often used in optimization problems to attain a solution which is less likely to be a result of overfitting.
No regularization
Without any regularization the solution to a linear regression is
$\beta=(X^TX)^{-1}X^TY$
L2 regularization
To understand regression it is much easier to first start with the more widely used L2 regularization, ridge regression. This is defined as
$\min_\beta (Y-X\beta)^T(Y-X\beta) + \lambda ||\beta||^2$
We can solve this to get a closed-form solution using a derivative
$0 = -2X^TY + 2X^TX\beta+2\lambda\beta$
$X^TY = (X^TX + \lambda I)\beta$
$\beta = (X^TX + \lambda I)^{-1}X^TY$
Let's analyze this result in contrast to the solution without regularization and see what it means. We can see that the only difference is the added regularization term $\lambda$ within the inverse. As we should know, the inverse of a larger value, will cause its result to get smaller. For example the inverse of 1/5, is larger than 1/2. So by adding the regularization term we actually reduced the associated weight $\beta$.
This is why regularizers are often called a penalty term. They further constrain the optimization function. By adding an additional cost associated to each weight, we can reduce the impact that each weight will have on our model. Thus reducing the potential for overfitting.
L1 regularization
L1 regularization is the penalty used for LASSO regression, it's cost function is defined as
$\min_\beta(Y-X\beta)^T(Y-X\beta) + \lambda||\beta||$
From what we learned above we can already tell that this additional cost will cause the resulting weights to be penalized. Unfortunately L1 regularization does not have a closed form solution because it is not differentiable when a weight $\beta$ falls to 0. Thus this requires some more work to solve. LASSO is an algorithm for finding the solution.
First let's remember that our constraint here is $||\beta||\leq c$ where c is a real number which is inversely related to $\lambda$. The plan is to find an appropriate $\lambda$ such that the minimum of the function falls within or on the contour of the constraint. This constraint has very sharp edges which lie on each dimensional axis at a distance $c$ from the origin. You can imagine it like a diamond in 2D space, an octohedron in 3D space, etc.
In high dimensional space these spikes have a very high likelyhood to being hit by the function you wish to optimize thus this will cause many of the features to have an associated weight of 0. For example if we compare the regularization of this line in 2D space using both L2 and L1 regularization.
In this picture the distance $r$ is equivalent to the factor $c$ I used above which is inversely related to the regularization term $\lambda$. The function being optimized (the cost function) is the described using the ellipses which represent their levels (like an altitude map). The shape at the center of the axis is the regularizer which is used to constrain the weights associated to each feature.
We can first notice that the original function is minimized exactly at its minimum. However, choosing this solution has a high probability of being one which overfits the training data.
We see that L2 regularization did add a penalty to the weights, we ended up with a constrained weight set. However both weights are still represented in your final solution. The function being optimized touches the surface of the regularizer in the first quadrant.
However, in the L1 case we can see that $\theta_2$ is the only feature that is required for this regression problem and $\theta_1$ will have a weight set to 0. Thus we can ignore it. It just happens that smooth functions encountering a function with pronounced edges tends to hit these edges with higher probability than hit the vertices. I do not know the proof for this, but I have seen it in a course as supplemental material and it was complicated. This is why L1 regularization is often used for feature selection.
Combining both together
What is often done is first using L1 regularization to find out what features have lasso weights which tend to 0, these are then removed from the original feature set. Then with this new feature set we can apply L2 regularization in order to still control these new features to control for overfitting.
