# Tag Info

7

Not every seed is the same. Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility: def seed_everything(seed=42): """" Seed everything. """ random.seed(seed) os.environ['PYTHONHASHSEED'] = str(seed) np.random.seed(seed) torch.manual_seed(seed) torch.cuda.manual_seed(seed) ...

7

F1 will never be zero, but very near to zero for a bad classifier. If TP or TN is zero then there isn't any need to check F1.

5

It's a mistake on Wikipedia. $F_{1}$ as the harmonic mean is defined only at positive real numbers. $PRE$ or $REC$ could be equal 0 in case $TP=0$. Which provides to undefined result $F_1=\frac{0}{0}$.

3

FP32 and FP16 mean 32-bit floating point and 16-bit floating point. GPUs originally focused on FP32 because these are the calculations needed for 3D games. Nowadays a lot of GPUs have native support of FP16 to speed up the calculation of neural networks. If you look at some benchmarks (https://blog.slavv.com/titan-rtx-quality-time-with-the-top-turing-gpu-...

3

Well, grid search involves finding best hyperparameters. Best according to what data set? a held out validation set. If that's what you mean by cross validation, then they necessarily happen simultaneously. It doesn't really make sense to do something called cross validation before testing hyperparams - indeed, what would you be evaluating? CV as in k-fold ...

2

Both are done together! Grid search is how you determine the hyperparameters of a model whereas, Cross-Validation is the process of running your model on a data separate from training datase to gauge your models performance on a particular hyperparameter. Grid search allows you to have some potential hyperparameters among which you compare the model's ...

2

I'm assuming that the "propensity model" predicts a customer's likelihood to take some desired action. If it's possible at your organization, then a good way to measure the model's effectiveness is an A/B test. Select a set of leads to act as a control group using a baseline selection method. Ideally, the baseline method would be whatever is currently in-...

2

It is always better to keep sample sizes close each other. The problem you are facing is Imbalanced Classification. There are lots of methods you can apply such as upsampling/downsampling, synthetic data generation (check SMOTE). Model: I would first convert the model to binary classification such that: model 1 predicts: A or not A model2 predicts B or ...

2

As you said, since the Koalas is aiming for processing the big data, there is no such overhead like collecting data into a single partition when ks.DataFrame(df). However, the overheads are occurred when creating a default columns for creating the _InternalFrame which internally manages the metadata between pandas and PySpark. Koalas is internally using ...

2

It can't be exactly zero. We need exactly one (only one) of precision. Or recall to be zero to make f1 = zero, but both have "tp" as the numerator. #### Will be Nan y_test = np.array([0,0,1,1]) y_pred = np.array([0,1,0,0]) from sklearn.metrics import confusion_matrix tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel() precision = tp/(tp+tn) recall = ...

1

Another useful "graph" is the Validation curve. This will show you the difference between your training curve and you testing curve. https://scikit-learn.org/stable/auto_examples/model_selection/plot_validation_curve.html#sphx-glr-auto-examples-model-selection-plot-validation-curve-py

1

The basics are: Precision-Recall Curve ROC Curve Then, you can plot other features of the model to get insights about generalization, learning process, overfitting/underfitting, like accuracy vs epochs, loss versus epochs, to name a few.

1

I think your confusion comes from the fact that the false negative of $M$ as you thought it, is not top-$k$ accuracy but perhaps $1$ $-$ top-$k$. Moreover, how do you evaluate the condition if g' in top-$k$ most similarity graphs with g? In any case, for your case I thought the following. Let's say that if the model $M$ returns a value $o > o^*$, where \$o,...

1

Your confusion seems to stem from this line: print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100)) The best_score_ is not exactly a training score (nor is it an unbiased estimate of future performance*): as you say, it's the averaged score across different fold splits, but each of the scores that get averaged are the performance of the ...

1

"I'm not sure is it reasonable?" yes it is. The metric you refer to is known as the Jaccard index.

1

I believe Kolas is the Databricks DF equivalent of a Python DF and the equivalent of a Spark DF (I think Kolas is very,very new; released just a few months ago). I don't know what you mean by cost, but you can easily switch between Spark DF and Pandas DF. See the examples below. # Convert Koala dataframe to Spark dataframe df = kdf.to_spark(kdf) # Create ...

1

For comparing two rankings Spearman's rank correlation is a good measure. It's probably worth a try, but since your gold truth appears to be binary I would think that top-N accuracy (or some variant of it) would be more appropriate (advantage: easy to interpret). You could also consider using the Area Under the Curve (AUC), using the predicted rank as ...

1

A 1/3 - 2/3 repartition is not that unbalanced. Your problem shouldn't require balancing. The train/test set partition seems to be done correctly, as it seems implied by checking data histograms. Doing that randomly is usually ok, and when it's not it will inflate your test performance with data leakage, which doesn't seems to be the case here. Imo the ...

Only top voted, non community-wiki answers of a minimum length are eligible