8

That is generally not true, to be more accurate we can say that L1 promotes sparsity. if a weight is larger than 1 then L2 cares more about it than L1 while if a weight is less than 1 then L1 cares more about it than L2. For a quick example imagine two weights, $w_1 = 15$ and $w_2 = 0.02$, let's imagine that the model considers reducing both of those ...


5

This is a very general question, however, there are many different solutions as explained below. For your case, probably, item 2 is not the case because you can not gather a large number of data points. I would recommend using solutions 1, 3, 5, and 6 (I see you used this method but try to combine it with other solutions such as cross-validation, ...


4

The convergence time is sensitive to the data you have and a random seed. Specifically, the convergence time is linear in expectation in all three cases. SGDClassifier uses the stochastic gradient descent for optimization. Since L1 loss is only subdifferential, the L1 penalty causes the algorithm to converge noticeably slower. Comparing with or without the ...


4

The way most people gain an initial understanding of label smoothing (and what most common explanations have to say on the subject) plays a great role in how one would approach this question. At first glance, label smoothing is exactly what the name suggests: we modify the labels or some portion of them in order to get a better, more general, more robust ...


2

Here I go with a worked example for answering mainly your first 2 questions, with some code based on this scikit-learn example. Let's generate a rough parabola as follows: import numpy as np import matplotlib.pyplot as plt def f(x): """ function to approximate by polynomial interpolation""" return np.square(x) # generate points used to plot x_plot ...


2

When you do linear regression you have to leave out one column as it's a singular matrix and hence columns are linearly dependent and we cannot calculate the inverse. But when you do regularization it take cares of singularity. The matrix is almost surely nonsingular. Hence we don't need to drop a column and if you drop different columns from each feature it ...


2

Penalty adds an additional term to the loss function. In your case, it made your model require more iterations to converge. When penalties are added, if you see the converged value of your original loss(log) only, it will be a lower value than the case with no penalty. Which shows the advantage, adding a penalty makes your model converge to lower loss.


2

When we implement penalized regression models we are saying that we are going to add a penalty to the sum of the squared errors. Recall that the sum of squared errors is the following and that we are trying to minimize this value with Least Squares Regression: $$SSE = \sum_{i=1}^{n}(y_i-\hat{y_i})^2$$ When the model overfits or there is collinearity present, ...


2

I am unsure there will be a formal way to show which is best in which situations as it depends on many factors like your dataset, architecture of the your ANN - simply trying out different combinations is likely best. It is worth noting that Dropout* is actually doing more than just regularization, it makes the model more robust,allowing it to try different ...


2

The teacher/student model approach (at least as I understand it) could be used. It is normally used to replace all layers, but there is nothing stopping you applying it to a subset of layers. First you create training data, by running your training data through the (trained) network, and recording the inputs to layer i, as your new training data (x) and the ...


1

Kaggle is a crowd source platform with no quality control. It is to be expected that there will be deviations from best practices.


1

Have a look at svc_cv.cv_results_: there are many values of C that tied for best, with accuracy 99.6%, and the chosen C is the smallest of those. The point is that the width of the margin doesn't affect the actual hyperplane very much, and so the accuracy score doesn't change very much. A few suggestions: With larger and less-separable data sets, this ...


1

@Ethan is correct about the formulation of the lasso penalty, and I think it's particularly important to understand it in that form (for one thing, because that same penalty can work with other models like neural networks, tree models, generalized linear models, ...). But, to your question: If $\lambda=0.5$ then does it mean that those coefficients whose ...


1

I support your "proof is in the pudding" sentiment. Some of those hyperparameters are not that extreme, in my experience. Boosted trees very often perform best with weak individual learners; your max_depth is right in line with what I'm used to seeing as best. The score regularization penalties (alpha, lambda) don't play as important a role in my ...


1

Exactly as it says in the documentation; it's $1/\lambda$, where $\lambda$ is the regularization strength.


1

Title question The answer to the title question is a pretty clear-cut "no." For any fixed dataset, taking sufficiently strong regularization will make the linear model essentially constant, presumably not the best model. Like most other things, there's a bias-variance tradeoff: increasing regularization weakens the model's ability to learn the ...


1

No - non-parametric methods only means that the method does not assume a function form of the data. There are non-parametric methods such as Random Forest that do not always overfit. In fact nonparametric methods could underfit, it could lack the ability to fit the training data. An example of this would be a decision stump.


1

Welcome to the community! There are points coming to my mind: Check the amount of data in each block and then their distribution. First experience might be due to the lack of enough data in reserved block (i.e. you literally just trained but did not validated resulting to a complete overfitting) or having a totally different distribution (i.e. the ...


1

In transfer learning there are two parameters which influence the basic setup to go for: Size of new dataset Similarity of new dataset to dataset of pre-trained model When your dataset is small the problem is that high capacity pre-trained models can easily overfit if you re-train too many layers. And since you re-trained multiple layers this could be an ...


1

There are generic methods to avoid overfitting, but I'd like to address your specific problem. Like you said, your dataset doesn't have a lot of examples compared to the number of features. This, on its own, increases the risk of overfitting, especially if you use a more complex model such as GradientBoost or RandomForest (I'm not sure I'd use either when ...


1

With sklearn you can have two approaches for linear regression: 1) LinearRegression object uses Ordinary Least Squares (OLS) solver from scipy, as Learning rate (LR) is one of two classifiers which have closed form solution. This is achieve by just inverting and multiplicating some matrices. 2) SGDRegressor which is an implementation of stochastic gradient ...


1

I think your best bet to answer that question is to continue to test it on other datasets to confirm that your observation holds true in multiple situations. Here is my intuition (although I'm by no means an expert). The regular loss function is (wx + b - y)^2 The loss function for L2 Regularization is L2 = (wx + b -y)^2 + lamdba * w^2 In the first ...


1

Backpropagation doesn't handle Regularization like this i.e. if you are thinking "10 weights make penalty 100, so 100 weights will make penalty 1000. So let's have a smaller $\lambda$ Backpropagation uses the partial differentiation of the Loss. Now the Loss has an extra $\Sigma$$w_i^2$ and so the derivative will have an extra piece which will be ...


1

Here's a link to a good answer for the follow up question of "should you use both L1 and L2 regularization terms?" Summarized briefly here: These lightGBM L1 and L2 regularization parameters are related leaf scores, not feature weights. The regularization terms will reduce the complexity of a model (similar to most regularization efforts) but ...


1

One possible solution when you do not have enough data is to use Transfer learning. This helps you to improve the performance of your model on the test data set. So, you can easily use one of the available pre-trained models in technical literature and update its weights based on your data. Take a look at this video. It is very helpful and you get a lot of ...


Only top voted, non community-wiki answers of a minimum length are eligible