15

Multiple papers have opined that only in rare cases is there a known distribution of the error as a function of the number of features and sample size. The error surface for a given set of instances, and features, is a function of the correlation (or lack of) between features. This paper suggests the following: For uncorrelated features, the optimal ...


9

From my own experience:In one case, I worked with a real database that is very small (300 images) with many classes, severe data imbalance problem and I ended up with using 9 features: SIFT, HOG, Shape context, SSIM, GM and 4 DNN-based features. In another case, I worked with very large database (> 1 M images) and ended up with using only HOG feature. I ...


8

Yours is not an example of nested cross-validation. Nested cross-validation is useful to figure out whether, say, a random forest or a SVM is better suited for your problem. Nested CV only outputs a score, it does not output a model like in your code. This would be an example of nested cross validation: from sklearn.datasets import load_boston from ...


8

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune. However, i've also read that model selection shoud be done before tuning the parameters. Before tuning you need to do some kind of pre-processing before tuning the parameters. Usually your pipeline will consist of: Get Data and Clean It. Do some EDA ( ...


7

This is a hard problem and researchers are making a lot of progress. If you're looking for supervised feature selection, I'd recommend LASSO and its variants. Evaluation of the algorithm is very straightforward with supervised learning: the performance of whichever metric you choose on test data. Two major caveats of LASSO are that (1) the selected ...


7

No, he actually says the opposite: One final note: I should say that in the machine learning as of this practice today, there are many people that will do that early thing that I talked about, and said that, you know...​ Then he says (the "early thing" he talked about): selecting your model as a test set and then using the same test set to report the ...


7

Nested cross validation estimates the generalization error of a model, so it is a good way to choose the best model from a list of candidate models and their associated parameter grids. The original post is close to doing nested CV: rather than doing a single train–test split, one should instead use a second cross-validation splitter. That is, one "nests" an ...


7

Is there any model in machine learning that does not have parameters? Yes. k-nearest neighbors is parameterless (there is only a single hyper-parameter $k$). If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters? Exactly: such models require no training at all. k-NN in ...


7

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect. Adding new features is not always beneficial, because you increase the dimension of your search space and thus make the problem harder. In your particular case the increased complexity overweight the added value from extra features. ...


7

Yes, autocorrelation in residuals is a problem, but this is essentially because it is a clear illustration that there was more learnable information in the process you are modelling but your model missed it. In the unlikely event that you have two equally performant models but one shows significant autocorrelation (you can test for this using the Durbin-...


6

I'm familiar with three main approaches: A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes, and so have 44 states for the hidden phoneme layer in a voice recognition model. Estimation. The number of states can often be estimated beforehand, perhaps ...


6

Choose model A, if autocorrelation is significant residuals="mistakes in predictions" should be completely random, i.e. follow White noise. Now if something is significantly autocorrelated it wont be truly random and the independent error model is incorrect and it wont be a robust variance estimator. Prefer model A How to measure significant ...


5

Linear Regression is used for predicting continuous variables. Logistic Regression is used for predicting variables which has only limited values. Let me quote a nice example which can help you make the difference between the both: For instance, if X contains the area in square feet of houses, and Y contains the corresponding sale price of those ...


5

I think this is a good approach in general. However: Fine-tuning your model (online learning) depends a lot on the algorithm and model how well this works. Depending on your algorithm it might be wise to retrain the whole thing Your sample space might change overtime. If you have enough data maybe retraining every few days/weeks/months over only the last ...


5

I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. I suggest you first look at this very similar post, where I discuss some techniques to convert categorical variables to numerical values. Since you provide ...


5

I do not think you can estimate the effect of a variable without adding it to the model. This is because the effect of a variable on the model's discriminatory power depends on the strength of association between the outcome variable and the new variable whether the new variable are collinear with some of the old variables You could in principle estimate ...


5

Did you evaluate the results in the training set? Or in the test set? Those results are outstandingly good! Suspiciously good. I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may ...


4

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by class imbalance through the false positive rate FP/(FP+TN). If the number of negatives is a lot larger, this could be a potential issue. In short, the Gini Coefficient has similar pros and cons as ...


4

There isn't a well established way of estimating the number of data points that you'll need. It's much more an art than a science. As you gain more experience, you'll learn some common sense lessons (in hindsight) along the way. For instance, you should never have more parameters than data points; if you're building random forests, you should not have more ...


4

It depends... but of course that answer gets you nowhere. He is some rule of thumb for model complexity: Learning from data - VC dimension "Very roughly" you need 10 data points for each model parameter. And number of model parameters can be similar to number of features.


4

I'll try to provide some insight which will hopefully help. Can a normal NN model the time connections the same way like a RNN/LSTM does when it is just deep enough? Every neural net gets better in theory if it gets deeper. For a regular NN to model time connections properly, you could use the last n time steps as your input and the n+1th time step as ...


4

The explanation is simple, assume you have the following values: True Positives (TP) = 1 True Negatives (TN) = 998 False Positives (FP) = 1 False Negatives (FN) = 1 Accuracy = (TP + TN) / (TP + TN + FP + FN) = 999/1001 = 0.998 Precision = TP / (TP + FP) = 1/2 = 0.5 Recall = TP / (TP + FN) = 1/2 = 0.5 In summary you have an unbalanced dataset i.e. the ...


4

Yes, it will impact because when you change the loss function, the numerical value of the loss function will change. So, this will change gradient values of the parameters during the back propagation. Therefore, change in loss function will impact the parameters.


4

I have personally only heard the letters being spelled out: "Here we see the R-O-C curve on this pronunciation model is far from optimal"


4

I have not heard of any model agnostic way to measure model complexity. There are several strategies but they are model dependant. You can tackle the problem using different families of models. For linear models you can count the number of nonzero parameters that is using. Number of features used for the prediction. For decision tree you can count the ...


4

What you want to achieve with this validation strategy is a robust estimate of what combination of hyperparameters is good enough for your final model, so: for each combination of hyperparameters, you carry out k trainings (as you ask with your last question), following this schema: source of info once you have this k-trained models (i.e. for one ...


3

R squared is one of many model diagnostics. One calculation of it is (1-var(residuals)/var(response variable)). Since the response variable is constant, you want to minimize the average squared residuals (the variance), which is identical to least squares. The issue with using only R-squared is that if you have a lot of variables in a model, you will likely ...


3

Have you read Janssens' dissertation "Outlier Selection and One-Class Classification"? He has a chapter on evaluation which may be of use. Have you thought about artificial generation of negative instances? I had to deal with OCC evaluation awhile back and never came upon an entirely satisfactory solution. As I recall, the basic problem boils down to the ...


3

Bit late to the party, but here are some heuristics. binary classification problem with 20 instances in each class, is there any upper limit on the number of features to use? For training of linear classifiers, 3 - 5 independent cases per class and feature are recommended. This limit gives you reliably stable models, it doesn't guarantee a good model (...


3

No, area under receiver operating characteristic (AUROC) is just one metric amongst very many possibilities, even assuming you just want to pick a standard approach. There are too many to list in a simple Stack Exchange answer. You can take a look at this list extracted from scikit learn documentantion on metrics for example, which is not by any means ...


Only top voted, non community-wiki answers of a minimum length are eligible