17 votes
Accepted

Any "rules of thumb" on number of features versus number of instances? (small data sets)

Multiple papers have opined that only in rare cases is there a known distribution of the error as a function of the number of features and sample size. The error surface for a given set of instances, ...
user avatar
  • 356
9 votes

Any "rules of thumb" on number of features versus number of instances? (small data sets)

From my own experience:In one case, I worked with a real database that is very small (300 images) with many classes, severe data imbalance problem and I ended up with using 9 features: SIFT, HOG, ...
user avatar
9 votes
Accepted

Which is first ? Tuning the parameters or selecting the model

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune. However, i've also read that model selection shoud be done before tuning the parameters. ...
user avatar
8 votes
Accepted

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

Yours is not an example of nested cross-validation. Nested cross-validation is useful to figure out whether, say, a random forest or a SVM is better suited for your problem. Nested CV only outputs a ...
user avatar
7 votes

How to compare the performance of feature selection methods?

This is a hard problem and researchers are making a lot of progress. If you're looking for supervised feature selection, I'd recommend LASSO and its variants. Evaluation of the algorithm is very ...
user avatar
7 votes
Accepted

On coursera what exactly does Andrew Ng say in videos Lectures 60 & 61 of machine learning?

No, he actually says the opposite: One final note: I should say that in the machine learning as of this practice today, there are many people that will do that early thing that I talked about, and ...
user avatar
  • 250
7 votes

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

Nested cross validation estimates the generalization error of a model, so it is a good way to choose the best model from a list of candidate models and their associated parameter grids. The original ...
user avatar
7 votes
Accepted

Do models without parameters exist?

Is there any model in machine learning that does not have parameters? Yes. k-nearest neighbors is parameterless (there is only a single hyper-parameter $k$). If such parameterless models exist, ...
user avatar
7 votes

Adding feature leads to worse results

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect. Adding new features is not always beneficial, because you increase the ...
user avatar
  • 1,470
7 votes
Accepted

Is autocorrelation of residuals a problem in machine learning?

Yes, autocorrelation in residuals is a problem, but this is essentially because it is a clear illustration that there was more learnable information in the process you are modelling but your model ...
user avatar
7 votes
Accepted

How can I choose the best machine learning algorithms from all kinds of algorithms?

There is a theoretical result called the "no free lunch theorem" which proves that there is no "best ML algorithm" in general. It's important to understand how an algorithm works ...
user avatar
  • 22.1k
6 votes
Accepted

How do scientists come up with the correct Hidden Markov Model parameters and topology to use?

I'm familiar with three main approaches: A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes,...
user avatar
6 votes
Accepted

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by ...
user avatar
  • 5,995
6 votes
Accepted

Is there any way to explicitly measure the complexity of a Machine Learning Model in Python

I have not heard of any model agnostic way to measure model complexity. There are several strategies but they are model dependant. You can tackle the problem using different families of models. For ...
user avatar
6 votes

Is autocorrelation of residuals a problem in machine learning?

Choose model A, if autocorrelation is significant residuals="mistakes in predictions" should be completely random, i.e. follow White noise. Now if something is significantly autocorrelated ...
user avatar
  • 5,321
5 votes

When to use linear or logistic regression?

Linear Regression is used for predicting continuous variables. Logistic Regression is used for predicting variables which has only limited values. Let me quote a nice example which can help you make ...
user avatar
  • 8,036
5 votes

Machine Learning models in production environment

I think this is a good approach in general. However: Fine-tuning your model (online learning) depends a lot on the algorithm and model how well this works. Depending on your algorithm it might be ...
user avatar
5 votes
Accepted

Why are RNN/LSTM preferred in time series analysis and not other NN?

I'll try to provide some insight which will hopefully help. Can a normal NN model the time connections the same way like a RNN/LSTM does when it is just deep enough? Every neural net gets better in ...
user avatar
5 votes

Choosing a model for dataset with categorical variables

I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. ...
user avatar
  • 4,016
5 votes
Accepted

Adding new variable to model

I do not think you can estimate the effect of a variable without adding it to the model. This is because the effect of a variable on the model's discriminatory power depends on the strength of ...
user avatar
5 votes

Interpretation of ROC AUC score

Did you evaluate the results in the training set? Or in the test set? Those results are outstandingly good! Suspiciously good. I think you tried your results in the training set only, so your ...
user avatar
4 votes

From development environment to production

There isn't a well established way of estimating the number of data points that you'll need. It's much more an art than a science. As you gain more experience, you'll learn some common sense lessons (...
user avatar
  • 4,059
4 votes

Any "rules of thumb" on number of features versus number of instances? (small data sets)

It depends... but of course that answer gets you nowhere. He is some rule of thumb for model complexity: Learning from data - VC dimension "Very roughly" you need 10 data points for each model ...
user avatar
  • 355
4 votes

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

Gini coefficient shouldn't be to my understanding a bad mertric for imbalanced classification, because it is related to AUC, which works just fine. Maybe it was gini impurity not coefficient. Check ...
user avatar
4 votes

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

Credit models do not do a great job of predicting individual defaults, and the error rates are usually high. That is, a fairly high proportion of dubious borrowers do not default. One can always ...
user avatar
  • 41
4 votes

can accuracy rise while precision and recall drop?

The explanation is simple, assume you have the following values: ...
user avatar
  • 2,156
4 votes

P-value mining on large number of combinations of variables

You need to investigate multiple hypothesis correction methods, like Bonferroni correction or Benjamini-Hochberg false discovery rate. The problem with this sort of analysis is that your associations ...
user avatar
4 votes
Accepted

Does the choice of error function impact the model parametrs?

Yes, it will impact because when you change the loss function, the numerical value of the loss function will change. So, this will change gradient values of the parameters during the back propagation. ...
user avatar
4 votes

How do you pronounce ROC?

I have personally only heard the letters being spelled out: "Here we see the R-O-C curve on this pronunciation model is far from optimal"
user avatar
  • 14k
4 votes

Why my network needs so many epochs to learn?

My opinion: You should try to increase the learning rate of your model (or even other parameters of your optimizer - e.g. momentum). To answer your questions: Why the network is still learning after ...
user avatar
  • 7,558

Only top scored, non community-wiki answers of a minimum length are eligible