21
There are a couple of nuances here.
Complexity question very important - ocams razor
CV - is this trully the case 84%/83% (test it for train+test with CV)
Given this, personal opinion: Second one.
Better to catch general patterns. You already know that first model failed on that because of the train and test difference. 1% says nothing.
13
It depends mostly on the problem context. If predictive performance is all you care about, and you believe the test set to be representative of future unseen data, then the first model is better. (This might be the case for, say, health predictions.)
There are a number of things that would change this decision.
Interpretability / explainability. This is ...
12
As a rule of thumb, removing outliers without a good reason to remove outliers rarely does anyone any good. Without a deep and vested understanding of what the possible ranges exist within each feature, then removing outliers becomes tricky. Often times, I see students/new hires plot box-plots or check mean and standard deviation to determine an outlier ...
8
Why not, because the risks outweigh the benefits.
It might work in images, where loss of pixels / voxels could be somewhat "reconstructed" by other layers, also pixel/voxel loss is somewhat common in image processing. But if you use it on other problems like NLP or tabular data, dropping columns of data randomly won't improve performance and you will risk ...
8
The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted.
Maybe not. It's true that 100% training accuracy is usually a strong indicator of overfitting, but it's also true that an overfit model should perform worse on the test set than a model that isn't overfit. So if you're seeing these numbers, something unusual is going ...
7
ML algorithms all learn a set of numbers associated with whatever model you're training. For a neural network, it's the weights on the network links. For regression, it's the coefficients. And so on. These numbers you can store and use later (most ML libraries have utilities for doing this). You use the coefficients to make predictions on data on which you ...
7
Tophat makes some great points. Another thing to consider is that you removed close to 20 percent of your data by removing the "outliers" which leads me to believe that they really aren't outliers, rather, just extreme values. Certainly, there may be an outlier on one dimension that you should look at, but with such a rich data set, an extreme value in one ...
7
The reason is kernel shap sends data as numpy array which has no column names.
so we need to fix it as follows:
def model_predict(data_asarray):
data_asframe = pd.DataFrame(data_asarray, columns=feature_names)
return estimator.predict(data_asframe)
Then,
shap_kernel_explainer = shap.KernelExplainer(model_predict, x_train, link='logit')
...
7
This is an implementation detail, and I wouldn't necessarily rely on this behavior, but presently in sklearn, it will choose the "first" class.
The predict method calls for the probability prediction, then takes the argmax, which in case of ties takes the first one:
https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/tree/_classes....
6
Good question and welcome to Datascience
Imagine you have the tree as follows.
Machine Learning Models
|
----------------------------------------------------
| |
Supervised Unsupervised
|...
6
There may be a few reason this is happening.
First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did you train your model on? You may have made a coding mistake and compared two same list.
Did you use different test set for testing? The high accuracy may be due ...
6
Looking at the training epochs, it seems to me you set a patience parameter that is too short. Please consider removing early stopping at all, for a model trained on 1500 observations only. Early stopping comes useful for particularly heavy models, but in this you shouldn't need it.
I think the size of each mini-batch is very small. That would make gradient ...
5
I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer "components", "individual/constituent models", or something like that.
If you definitely want to find a "correct" term, one way to discover it (if it exists) is to look into an ensemble learning early paper.
To arrive at one, a good way is to search for a ...
5
It is not uncommon to use dropout on the inputs. In the original paper the authors usually use dropout with a retention rate of 50% for hidden units and 80% for (real-valued) inputs. For inputs that represent categorical values (e.g. one-hot encoded) a simple dropout procedure might not be appropriate.
They also argue that dropout applied to the inputs of ...
5
I'm not very sure what you mean by "60% accuracy using AUC". Accuracy and AUC are two different metrics... I'm going to answer as if you're referring to classification accuracy, since that's in your title and the first sentence of your post.
First of all, don't use accuracy to evaluate performance on imbalanced data!
Your dataset has an imbalance ratio of ...
5
Before I answer your question, let me tell you this, You can go on and train a model from scratch, but you will definitely end up using one of the object detection architectures, be it Mask R-CNN, Faster R-CNN, Yolo or SSD. Your problem is a simplified version of what these architectures are trying to solve. These are generic object detectors that some of ...
5
The number of parameters is 60,344,232, according to:
https://resources.wolframcloud.com/NeuralNetRepository/resources/ResNet-152-Trained-on-ImageNet-Competition-Data
Although, from https://arxiv.org/pdf/1512.03385.pdf (page 6), it looks like an ensemble was used for ILSVRC 2015:
"We combine six models of different depth to form an ensemble (only with two ...
4
The explanation is simple, assume you have the following values:
True Positives (TP) = 1
True Negatives (TN) = 998
False Positives (FP) = 1
False Negatives (FN) = 1
Accuracy = (TP + TN) / (TP + TN + FP + FN) = 999/1001 = 0.998
Precision = TP / (TP + FP) = 1/2 = 0.5
Recall = TP / (TP + FN) = 1/2 = 0.5
In summary you have an unbalanced dataset i.e. the ...
4
The default hyper-parameters of the DecisionTreeClassifier allows it to overfit your training data.
The default min_samples_leaf is 1. The default max_depth is None. This combination allows your DecisionTreeClassifier to grow until there is a single data point at each leaf.
Since you are having $100\%$ accuracy, I would assume you have duplicates in your ...
4
This comes down to the change-of-base formula. For any two numbers $a$ and $b$, the following equation is true.
$$
\log_a(x) = \frac{\log_b(x)}{\log_b(a)}.
$$
What this means is that the errors are proportional. So if you wanted to change to using $\log_{10}$, you would simply end up multiplying by a constant factor, and model selection would be the same.
...
4
I think you should consider time series modeling instead of observation based classification models. In the letter, you are propagating error in each prediction year.
I would use ARIMA, LSTMs, maybe semi-supervised models and motif discovery techniques.
4
I suggest to try a log transformation. This has two potential benefits:
The range of x values becomes smaller
Your transformed data might be closer to resemble a normal distribution (only relevant for some models, e.g. not for trees)
Here are two toy examples to illustrate:
Toy example 1
s = np.random.lognormal(3, 1, 1000)
plt.hist(s, 100)
plt.show()
...
4
Author of the paper here - I missed that this is apparently not a TensorFlow function, it's equivalent to Sonnet's scale_gradient, or the following function:
def scale_gradient(tensor, scale):
"""Scales the gradient for the backward pass."""
return tensor * scale + tf.stop_gradient(tensor) * (1 - scale)
4
This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling.
Here are a few links:
Lookalike at LinkedIn
Lookalike at Pinterest
Lookalike at Yahoo
Another lookalike from Yahoo
Academic paper on lookalike (not sure where the authors work)
Google's lookalike patent (this one is a lot ...
4
I have not heard of any model agnostic way to measure model complexity. There are several strategies but they are model dependant.
You can tackle the problem using different families of models.
For linear models you can count the number of nonzero parameters that is using. Number of features used for the prediction.
For decision tree you can count the ...
3
It depends on the ensemble model technique. If you are going to use a bagging approach then the term individual models or alternative models is the appropriate. But in the case of boosting methods the appropriate term is weak learner (classifier). Boosting algorithms is a family of machine learning algorithms that convert weak learners to strong ones. A weak ...
3
I have heard people calling them "weak learners" many times, but this is only when they are not very acurate themselves.
3
1) Supervised learning is most of the time the process of learning a mapping, e.g relation, of input features x (sample) to an output y (often labels).
Unsupervised learning doesn’t not use labels /output y to learn a relation between the samples and possible labels (ex: clustering).
2) Classification and regression are two types of supervised learning (...
3
Looks like a special case for incremental learning / life-long learning. There are many papers in this line of work:
https://arxiv.org/abs/1611.07725
https://openreview.net/forum?id=BkloRs0qK7
Most try to preserve the model parameters in a way that reduces the loss of previous knowledge ("catastrophic forgetting"), but some also try to find (or keep) some ...
3
I've been in that situation before,
there's this article on medium where the guy uses keras,tf for predicting credit card fraud detection using autoencoders which have Dense layers, but you can try the same with LSTM, can't say for sure whether it will work, but if in case it doesn't work, please try Conv1d because nowadays convolutional networks are more ...
Only top voted, non community-wiki answers of a minimum length are eligible
