Questions tagged [performance]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
16 views

Create new performance indicators (error metrics)

I am wondering if any of you happen to know of a procedure/approach/rationale to develop new performance indicators (error metrics) that can be used to evaluate the prediction capability (say, ...
2
votes
0answers
11 views

correlated variables & model performance: optimal trade-off

on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift. assuming little to medium ...
2
votes
0answers
40 views

Creating a new feature from an existing one using decision trees

Is it possible to create a new feature out of two, or more than two existing features using a decision tree? If so, how, and can it produce features with good information value that can better help ...
0
votes
0answers
12 views

Does flipping binary class labels impact performance metrics?

I am working on a classification problem with 0 and 1 class labels. I am wondering if I flip the class labels (i.e if I label all the formerly 0 classes as 1 and all the formerly 1 classes as 0) how ...
0
votes
0answers
28 views

2D Z-score/Mahalanobis distance that includes a penalty for uncertainty

I have some 2D points and I want to assess their performance against the target point. When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
1
vote
1answer
24 views

KNN efficient implementation

The KNN algorithm is very handy and particularly suited to some of my problems, but I can't find any resources on how to implement it in production. As a comparative example, when I use a neural ...
0
votes
0answers
12 views

What is a reasonable standard deviation in accuracy between different train-test splits?

My model's performance depends on the train-test-split performed. I did 1000 train-test splits and had an average accuracy of 75.4 % and the accuracy had a standard deviation of 2.4 % over those 1000 ...
0
votes
0answers
23 views

How to increase accuracy and decrease loss of my model

https://jovian.ai/casella0798/badmodel I created the model above to predict red wine quality. I have 6 classes, from 3 to 8. Dataset is unbalanced, with a lot of classes 5 and 6. My model performs ...
2
votes
2answers
39 views

Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set?

I know that choosing between models produced by one algorithm with different hyperparameters the metric for choosing the best one should be the cross-validation on train set. But what about choosing ...
2
votes
2answers
36 views

Once a predictive model is in production, how it can be evaluated?

I have a data science project, predicting customer's next purchase day. Customer's one year behavioral data was split to 9 and 3 months for train and test, using RFM analysis, I trained a model with ...
0
votes
1answer
224 views

CNN inference is slow on Jetson Nano

I'm running what I believe is a pretty lightweight CNN on an nVidia Jetson Nano with Jetpack 4.4. nVidia claims the Nano can run a ResNet-50 at 36fps, so I expected my much smaller network to run at ...
0
votes
0answers
20 views

When to use AUROC OvR vs. AUROC OvO?

For multiclass problems, there are 2 versions of the AUROC metric: the AUROC OvR and AUROC OvO. Does anyone know in what particular cases we would use AUROC OvR vs. AUROC OvO? In the general academic ...
1
vote
1answer
19 views

How do data types influence hardware (CPU / GPU / TPU) performance?

I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
0
votes
0answers
11 views

Classification model performance - metrics for getting number in each class correct?

I'm fairly new to predictive modelling, so apologies if this is a stupid question. I am working on a classification problem (predicting if customers commit fraud or not), and have been comparing a few ...
1
vote
1answer
74 views

What is the appropriate statistical significance test for multi-class classification?

I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have ...
2
votes
0answers
27 views

Fast PR / ROC curves and corespondings AUPR / AUROC

I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC). Its is quite easy to perform those ...
1
vote
0answers
25 views

How to balance time/effort with transformations, feature selection, and models efficacy in nlp? [closed]

Edit: Question has been edited for reopening (see comment section for justification) Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times ...
1
vote
1answer
25 views

Business-related metric for an optimization problem

We have an optimization problem on hand that is the following. Let's say we have 10 different treatments that we might offer, all of them are equally good for us, but people have different propensity ...
1
vote
2answers
56 views

Is fitting two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set more performant than one with 1000?

If I fit two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set, would it have better results than fitting a RandomForestClassifier with 1000 trees and ...
3
votes
3answers
771 views

Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

Suppose I've trained a RandomForest model with 100 trees. I then have two cases: I drop the first tree in the model. I drop the last tree in the model. Would the model performance be less in the ...
0
votes
0answers
8 views

Is iterative pruning always structured?

Iterative pruning works like the following: Evaluate model performance, continue if performance is fine Apply pruning Train the model Go back to step 1 When doing this process using unstructured ...
1
vote
2answers
38 views

Graphs for representing performance of machine learning classifiers [closed]

Can any please guide about how many types of the graph other than ROC can be plotted to represent the performance of the machine learning classifier?
5
votes
3answers
74 views

Finding out why your model is doing better?

I fitted a logistic regression model on a data set and got an AUC score of .70. I added some additional out-hot encoded categorical features to the model and the AUC improved slightly to .74. How do I ...
0
votes
1answer
24 views

How to derive false positive and false negative from top-k accuracy?

I am working on the following "equality identification" problem and become quite confused on how to reasonably define false positive and false negative in my case. Problem: Suppose I have a ...
1
vote
0answers
32 views

How to evaluate the performance of a model in production when labeling data is costly?

I have come to a problem for which I can't find a solution. Let's talk about a hypothetical binary classification problem in which you have some years of (human) labeled data. The final objective is ...
0
votes
0answers
14 views

Model performance metrics

I have a dataset with multiple numeric input values and a categorical output. How can I measure model performance with different algorithms. As the results are categorical, we can not obtain r squared ...
0
votes
2answers
163 views

Getting lower performance metrics when using GridSearchCV

I have defined an XGBoost model and would like to tune some of its hyperparameters. I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training ...
0
votes
1answer
125 views

How to run list comprehensions on GPU?

Is there a way to run complex list comprehensions like the following on GPU? ...
1
vote
1answer
3k views

FP16, FP32 - what is it all about? or is it just Bitsize for Float-Values (Python)

What is it all about FP16, FP32 in Python? My potential Business Partner and I are building a Deep Learning Setup for working with time series. He came up with "FP16 and FP32" while finding a GPU. It ...
8
votes
3answers
4k views

Can the F1 score be equal to zero?

As it is mentioned in the F1 score Wikipedia, 'F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0'. What is the worst condition that was mentioned? Even if we ...
0
votes
1answer
640 views

LSTM evaluation metric MAE explanation

I have a hard time understanding the LSTM model performance as I summarize my model as follow: ...
-1
votes
1answer
65 views

What is evaluation metric for two sets? [closed]

I've two sets one is ground truth and other is output of my machine learning models. Assume my groundtruth set is A={1,2,3,4,5} and output of machine learning model is B={3,4,5,6,7,8}. One way I can ...
3
votes
2answers
329 views

What is done first, cross validation or grid search?

When I have the data set to train a model with SVM, which procedure is performed first, cross validation or grid search? I have read this in a couple of books but I don't know in what order all this ...
1
vote
0answers
32 views

Improve performance of my CNN model

I am working on an image classification problem. There are 876 images in the training and 600 in the test dataset. It is a multi class classification for plants. Since this is my first CNN problem, ...
2
votes
0answers
159 views

Python library to process large files [closed]

I'm looking for a Python library to process large files without performance problems. For example I want to transfer large files with hash calculating. Python faces memory problems with such processes....
1
vote
3answers
998 views

Is there a cost associated with converting Koalas dataframe to Spark dataframe?

I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
1
vote
1answer
33 views

what metrics to evaluate rank order results?

I have searched on stackexchange and found a couple of topics like this and this but they are not quite relevant to my problem (or at least I don't know how to make them relevant to my problem). ...
2
votes
0answers
37 views

Multiclass Classification Task - Performance on Each Class Compared to Chance?

As a part of a classification task, a classifier has decided whether different books belong to class A, B or C (which are imbalanced) by looking at certain feature of the book. I have calculated ...
7
votes
1answer
3k views

Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67. I had the same algorithms and same set of hyperparamters during yesterday and ...
2
votes
3answers
72 views

Is over-fitting a matter of features engineering or a matter of parameters set?

I am using sklearn package to make models. I tried randomly to set some paramater to a sklearn.ensemble.RandomForestClassifier ...
7
votes
2answers
116 views

Which statistical test tells which classifier performs better than the other?

I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do ...
0
votes
1answer
27 views

Customer Intelligence - How to measure success?

we are creating models that aim to filter new leads from our current customer base. We started to create propensity models that calculate a percentage for each customer for a certain product group. I ...
3
votes
2answers
120 views

Does Sampling size matters in Multi classification Model

I am working on a multi class classification model where few of the class are with less data compare to other classes. I used random sampling technique to create a sample from the population keeping ...
1
vote
1answer
47 views

Relating ROC curves with class statistics

I have three neural net models that I am running on the same dataset (of 7 classes) and calculate their class performance and also ROC curves. The firs tmodel is a 4-layer model with 8 neurons in each ...
1
vote
1answer
45 views

Faster Data Transfer?

I'm doing a project which involves a small dataset (4GB). I'm trying to upload it to Paperspace to do some analysis but it's taking an absurd amount of time. Using Gradient, it would upload maybe ...
2
votes
1answer
40 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
1
vote
0answers
19 views

Evaluating randomness in a model performance

I'm evaluating the variability in performance (AUC) in the test set of a machine learning model with an intrinsic random component (xgboost). How many sources of variation should I use? Just ...
2
votes
2answers
981 views

How label smoothing and label flipping increases the performance of a machine learning model

I have seen posts and research papers mentioning these techniques for improving the performance of a machine learning model. These techniques certainly make some sense in the case when we are not ...
1
vote
2answers
183 views

AUC ROC Threshold Setting in heavy imbalance

I am doing binary logistic regression on a dataset with very heavy class imbalance. Class 1 is only 1% of data. When I train logistic regressor without class weights I get ROC AUC Score of 0.6269. ...
2
votes
1answer
783 views

Xgboost multiple class predictive performance beats one versus rest

I have an NLP task I'm tackling with xgboost (R implementation). Before describing my doubt I'll give you some background: I have a corpus of documents for which I did topic discovery, using a term ...