Questions tagged [performance]
For Question about Performance of a data science, statistical or machine learning model. Performace is a direct way to measure the efficiency of model. The Performance measure deals with time, accuracy and scalability for improve the model.
177 questions
2
votes
0
answers
26
views
Compare classification performance of dataset subsets based on factors
Let's say I have a dataset like this on which I want to perform classification:
id
feature
class
factor
1
...
1
A
2
...
1
B
3
...
2
A
4
...
2
B
$\vdots$
How can I compare the performance of a ...
- 21
2
votes
0
answers
11
views
Why might transforming my features improve the performance on a simple decision tree?
The features & target in my dataset are very skewed. Could anyone explain why transforming the features & target (I'm using a Yeo-Johnson transformation) is significantly improving the ...
- 21
0
votes
0
answers
208
views
In my case XGBoost is faster than LightGBM. How do I achieve better speed?
I hope that this place is the right one to ask the following question:
I'm training a machine learning model.
The train size is 85000x55. 49x ...
- 101
2
votes
1
answer
154
views
What is appropriate Individual KPI for AI projects?
I work in the sales department of electronics component manufacturing company and we do data science projects using traditional algorithm like Random forests (success likelihood of design project), ...
- 2,655
0
votes
0
answers
30
views
What is the most accurate way of computing the evaluation time of a neural network model?
I am training some neural networks in pytorch to use as an embedded surrogate model. Since I am testing various architectures, I want to compare the accuracy of each one, but I am also interested in ...
- 1
1
vote
1
answer
40
views
How to evaluate the performance of a prediction model across multiple predictions of the same event?
I was thinking of a hypothetical situation where you have a prediction model that can be used to predict the winner of an upcoming football match between Team A and Team B.
Say for the sake of the ...
- 13
0
votes
1
answer
75
views
Why does precision decrease with inceasing threshold?
I've trained a Logistic Regression model using scikit-learns LogisticRegression class. I'm dealing with stock data so it's quite noisy and difficult to predict ...
0
votes
0
answers
17
views
Does it make sense to do hp tuning for a Random Forest for top k precision or recall?
I've trained an RF with a binary classification task that achieves mediocre performance. However, they way it is intended to be used would have end-users look only at predictions with high scores (...
0
votes
0
answers
15
views
Passing the Parallel API tests in PettingZoo for custom multi-agent environment
from pettingzoo.test import (
parallel_api_test,
parallel_seed_test,
max_cycles_test,
performance_benchmark,
)
I have a custom multiagent ...
- 121
1
vote
1
answer
154
views
Seeking guidance on understanding graphics card parameters for deep learning training
I am currently in the process of purchasing a new Nvidia graphics card for training deep learning models, and I have a few questions regarding the parameters involved and their relationship to the ...
- 11
0
votes
0
answers
117
views
This model is too slow. I'm looking for a good, fast-enough, out-of-the-box, pre-trained image classifier. Any tip?
I have been using this on a laptop without a GPU: https://github.com/pharmapsychotic/clip-interrogator
Currently it takes about 10s to classify a single image on my own computer.
I use ...
- 101
0
votes
1
answer
1k
views
How to reduce the false positives to improve the models performance?
I am currently building a binary classification model to predict order return rates. I used the GradientBoostingClassifier for training the model and also performed hyperparameter tuning using ...
1
vote
1
answer
69
views
Imbalanced performance metrics in binary classification
I am developing a binary classification model using sklearn pipeline for preprocessing and a soft voting classifier (Adaboost and Extratrees with 50 estimators). The dataset (3 million rows) contains ...
- 23
0
votes
1
answer
20
views
Applying the model on validation data achieves higher performance than on test set. Is this possible?
I trained a binary cross-validated classification model and got high performance (about 90) on the test data but when I apply the model to new unseen data to see how to performs, i get even higher ...
- 11
0
votes
1
answer
24
views
Total Retention Rate Calculated from Categories
I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working.
I am comparing the last 14 days (let's call it Period 1) to ...
0
votes
1
answer
80
views
Quantifying the performance of Stepwise Regression ran on Monte Carlo generated datasets & comparing them to your method of interest
The source data files and scripts referenced here and from whom lines of code are included here can be found in my GitHub Repository for this collaborative research project exploring the properties of ...
- 167
0
votes
1
answer
610
views
Why does Adam outperform SGD in logistic regression?
I am training a logistic regression model. In case it matters, the features are 1376-dimensional embeddings output from a neural network. I tried both SGD and Adam with a learning rate of $10^{-3}$ ...
- 113
0
votes
1
answer
157
views
Ordering training text data by length
If I have text data where the length of documents greatly varies and I'd like to use it for training where I use batching, there is a great chance that long strings will be mixed with short strings ...
1
vote
1
answer
95
views
How do well informed labels for ordinal encoding improve model performance?
From Kaggle's intermediate machine learning tutorial, it was stated that
for each column, we randomly assign each unique value to a different integer. This is a common approach that is simpler than ...
- 113
1
vote
1
answer
77
views
Overall acurracy +/- E (with 90% C.I.)
I am assessing the accuracy of my classification model. I performed a 4-folds cross-validation and I obtained the following Overall Accuracy: OA = (0.910, 0.920, 0.880, 0.910). So, the average OA is 0....
- 63
0
votes
1
answer
83
views
Do I need to use AUPRC for reporting classification results on an imbalanced dataset when the model was trained using upsampling and CV
I am working on a binary classification problem which dataset has about 5% of positive class samples. I split the dataset, 70% for training and 30% for testing. I used the test data only once for ...
- 1
0
votes
1
answer
698
views
precision and recall at k for movielens dataset
I wanted to recreate a very simple collaborative filtering example with the 1M movielens dataset I have from Kaggle (https://www.kaggle.com/datasets/odedgolden/movielens-1m-dataset) and then ...
- 183
1
vote
1
answer
34
views
Doubt about comparison of Machine Learning algorithm
I read this article about the comparison of Machine Learning algorithm.
According to this article, there is some characteristics that define the ML models performance. I know that there are some ...
- 491
0
votes
1
answer
622
views
Which classification_report metrics are appropriate to report/interpret for a binary label? Individual or macro average for both classes? scikit-learn
First, please forgive my ignorance; I am a newbie but dedicated to learning more.
Example: I have a using a random forest classifier to predict a binary outcome. The binary outcome equals 1 if people ...
2
votes
1
answer
2k
views
Imbalanced data set with Sample weighting - How to interpret the performance metrics?
Consider a binary classification scenario whereby the True class (5%) is severely outbalanced to the False class (95%). My data set contains numeric data. I am using SKLearn and trying some different ...
- 129
1
vote
1
answer
143
views
The behavior of the cross validation error and training error in underfitting case is not clear
I currently study the "Machine Learning" course on Coursera.org by Andrew Ng, it comes to a topic that discusses the performance of learning algorithms under different conditions.
Here, we ...
- 13
4
votes
1
answer
79
views
What could be good Perfomance Evaluation Metrics for a Data Scientist?
Background
I'm a Data Scientist and am being asked to come up with a set of metrics/KPIs to assess my annual performance, and things like bonuses (and in the worst case being fired) depend on that. ...
1
vote
0
answers
101
views
How to implement kfold and cv into Hybrid feature selection and evaluate the classification model performance?
I have been working on a Hybrid feature selection combined with hyperopt package for hyperparameter tuning and I am thinking about evaluating the performance of several model classifiers. I looked ...
- 111
0
votes
1
answer
765
views
Is sensitivity the same as recall in multiclass classification?
In Wikipedia, it is stated "In binary classification, recall is called sensitivity" under the Recall section. Are they both different in case of multi-class classification?
2
votes
1
answer
1k
views
How to run hdbscan clustering faster?
I'm using hdbscan to cluster embedding output from BERT, which took in a data file of >150k chat messages. The embedding process took a little over 4 minutes, but as of this writing the hdbscan ...
- 121
0
votes
0
answers
32
views
What should be the ratio between training time and accuracy?
What should be the ratio between time and accuracy?
I mean when should you drop the accuracy a little bit but it will take less time for the Classifier/Regressor to run?
Edit: As part of my studies I ...
- 33
1
vote
1
answer
45
views
Measure performance of classification model for training on different snapshots
I am trying to do binary classification on some chronological data. Let's assume we have weekly data from the first week of 2017 through the last week of 2020. Now we have found out that 26 weeks of ...
- 189
1
vote
1
answer
20
views
How to compute performance of a detection-classification system?
I use a yolo (y) to detect only one object and a multiclassifier (mc) that classifies that object.
Now, the problem is: what I have to do with yolo's false positive and false negative, if I want to ...
- 13
-1
votes
1
answer
737
views
nested cross validation vs. train-test split
I am trying to understand the main benefits of conducting a nested cross-validation compared to a simpler train-test split. Let us say I would like to build a prediction model. I initially split my ...
- 103
2
votes
1
answer
26
views
Performance measurement of an event extraction system
I have developed an event extraction system from text documents. It first clusters the data corpus and extracts answers for what, when and where questions. Final answers are determined by using a ...
0
votes
1
answer
565
views
What is the Most Efficient Tool in Python for row-wise manipulation of data?
I'm doing a lot of work that requires operations to be performed across rows, using the data in that rows's columns on other columns in the row.
I recently had to do some processing on a 1.2 million ...
- 1
2
votes
1
answer
28
views
Create new performance indicators (error metrics)
I am wondering if any of you happen to know of a procedure/approach/rationale to develop new performance indicators (error metrics) that can be used to evaluate the prediction capability (say, ...
- 121
1
vote
0
answers
27
views
correlated variables & model performance: optimal trade-off
on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift.
assuming little to medium ...
- 169
1
vote
0
answers
69
views
Creating a new feature from an existing one using decision trees
Is it possible to create a new feature out of two, or more than two existing features using a decision tree?
If so, how, and can it produce features with good information value that can better help ...
0
votes
0
answers
192
views
2D Z-score/Mahalanobis distance that includes a penalty for uncertainty
I have some 2D points and I want to assess their performance against the target point.
When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
2
votes
2
answers
269
views
KNN efficient implementation
The KNN algorithm is very handy and particularly suited to some of my problems, but I can't find any resources on how to implement it in production.
As a comparative example, when I use a neural ...
- 21
0
votes
0
answers
74
views
How to increase accuracy and decrease loss of my model
https://jovian.ai/casella0798/badmodel
I created the model above to predict red wine quality. I have 6 classes, from 3 to 8. Dataset is unbalanced, with a lot of classes 5 and 6.
My model performs ...
- 229
2
votes
2
answers
176
views
Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set?
I know that choosing between models produced by one algorithm with different hyperparameters the metric for choosing the best one should be the cross-validation on train set.
But what about choosing ...
- 157
2
votes
2
answers
87
views
Once a predictive model is in production, how it can be evaluated?
I have a data science project, predicting customer's next purchase day. Customer's one year behavioral data was split to 9 and 3 months for train and test, using RFM analysis, I trained a model with ...
- 23
1
vote
1
answer
1k
views
CNN inference is slow on Jetson Nano
I'm running what I believe is a pretty lightweight CNN on an nVidia Jetson Nano with Jetpack 4.4.
nVidia claims the Nano can run a ResNet-50 at 36fps, so I expected my much smaller network to run at ...
- 121
2
votes
1
answer
178
views
How do data types influence hardware (CPU / GPU / TPU) performance?
I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
- 2,444
2
votes
1
answer
1k
views
What is the appropriate statistical significance test for multi-class classification?
I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have ...
- 1,197
4
votes
0
answers
234
views
Fast PR / ROC curves and corespondings AUPR / AUROC
I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC).
Its is quite easy to perform those ...
- 2,444
1
vote
0
answers
26
views
How to balance time/effort with transformations, feature selection, and models efficacy in nlp? [closed]
Edit: Question has been edited for reopening (see comment section for justification)
Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times ...
- 141
2
votes
1
answer
40
views
Business-related metric for an optimization problem
We have an optimization problem on hand that is the following.
Let's say we have 10 different treatments that we might offer, all of them are equally good for us, but people have different propensity ...
- 21