Not every seed is the same.
Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility:
os.environ['PYTHONHASHSEED'] = str(seed)
According to me, it is not correct to co-relate loss with accuracy.
Loss is used to optimize the hypothesis such that we can get best
weights, accuracy is used to identify how well model is doing in term
of correctly predicting the values.
Model internally takes the reference of predict_proba() and returns 1 if probability is > .5 (for 1) , otherwise ...
A couple suggestions
MAE represents your mean error. This is essential to recognize as having an error of even half of your MAE on the low end of your spectrum ($34,900) is huge. However, to your point, at the high end of your spectrum it is quite small.
To solve the above, you should plot a graph to identify the patterns in your error. You'll plot your ...
The test of your model should always be done exclusively on unseen data. That is fundamental to asses your model's capacity to generalize and predict observations it has never seen.
If you test it using already seen data, that is like cheating, and overfitting will be 100% guaranteed.
Repeat the test only on unseen data, and check the difference between ...
I'm assuming that the "propensity model" predicts a customer's likelihood to take some desired action.
If it's possible at your organization, then a good way to measure the model's effectiveness is an A/B test. Select a set of leads to act as a control group using a baseline selection method. Ideally, the baseline method would be whatever is currently in-...
It is always better to keep sample sizes close each other. The problem you are facing is Imbalanced Classification.
There are lots of methods you can apply such as upsampling/downsampling, synthetic data generation (check SMOTE).
I would first convert the model to binary classification such that:
model 1 predicts: A or not A
model2 predicts B or ...
According to this guy, he got a 15x increase from Intel i7 to GeForce 1070.
You also may consider using AWS. You can use a machine 100x as powerful (as a single 1070) and your employer may find it attractive because the upfront sunk cost is zero.
For comparing two rankings Spearman's rank correlation is a good measure. It's probably worth a try, but since your gold truth appears to be binary I would think that top-N accuracy (or some variant of it) would be more appropriate (advantage: easy to interpret). You could also consider using the Area Under the Curve (AUC), using the predicted rank as ...
A 1/3 - 2/3 repartition is not that unbalanced. Your problem shouldn't require balancing.
The train/test set partition seems to be done correctly, as it seems implied by checking data histograms. Doing that randomly is usually ok, and when it's not it will inflate your test performance with data leakage, which doesn't seems to be the case here.
Imo the ...
Label flipping is a training technique where one selectively manipulates the labels in order to make the model more robust against label noise and associated attacks - the specifics depend a lot on the nature of the noise. Label flipping bears no benefit only under the assumption that all labels are (and will always be) correct and that no adversaries exist. ...
It depends on the type of dataset you have. For instance, if you are trying to classify different types of flowers, the order in which you train the model on the features is irrelevant.
However, that being said, if you are dealing with non-stationary data such as time series values i.e. stock market prediction, the order of the features relative to time, ...
My guess is that the reduction in performance is due to differences in versions PyTorch. The published benchmark uses torch==0.4.1 (even though the README states something different). You are using PyTorch 1.0.1.
PyTorch 1.0 speed is sometimes slower than lower than 0.4. PyTorch 1.0 can be speed-up by adding torch.backends.cudnn.benchmark = True and ...
If the training loss keeps improving while the validation loss stagnates or decreases is a sign of overfitting.
It means that your model is continuing to learn patterns in your training data, so the layer/unit count is certainly not too low. But these patterns are not not general and does not exist in your validation data.
To combat this you can increase ...