Questions tagged [performance]

For Question about Performance of a data science, statistical or machine learning model. Performace is a direct way to measure the efficiency of model. The Performance measure deals with time, accuracy and scalability for improve the model.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

How to optimize Pandas DataFrame reorganizing?

I have a DataFrame which looks like this: date person value 2022-05-01 A 5 2022-05-01 B 4 2022-05-02 A 5 2022-05-02 B 9 I want to convert it to that form: ...
user avatar
0 votes
0 answers
11 views

Inference speed of ReLU networks

I'm fairly new in the topic, and I was wondering whether some of you can point to existing works in which the inference of deep neural networks with ReLU activation functions is tested on GPUs as a ...
user avatar
0 votes
0 answers
6 views

Detect change in performance of new rasa model

I have a rasa nlu model running to detect intents and entities. Whenever a new model is loaded by rasa after training, I want to know how good or bad it is from previous model. What I am planning is , ...
user avatar
1 vote
1 answer
22 views

Doubt about comparison of Machine Learning algorithm

I read this article about the comparison of Machine Learning algorithm. According to this article, there is some characteristics that define the ML models performance. I know that there are some ...
user avatar
  • 411
0 votes
0 answers
7 views

What is the impact of an outlier in a dependent variable and independent variable on model performance?

What is the impact of an outlier in a dependent and independent variable on model performance in regression and other machine learning models?. Is outlier in dependent variable more impactful than in ...
user avatar
  • 109
0 votes
1 answer
48 views

Which classification_report metrics are appropriate to report/interpret for a binary label? Individual or macro average for both classes? scikit-learn

First, please forgive my ignorance; I am a newbie but dedicated to learning more. Example: I have a using a random forest classifier to predict a binary outcome. The binary outcome equals 1 if people ...
user avatar
2 votes
1 answer
153 views

Imbalanced data set with Sample weighting - How to interpret the performance metrics?

Consider a binary classification scenario whereby the True class (5%) is severely outbalanced to the False class (95%). My data set contains numeric data. I am using SKLearn and trying some different ...
user avatar
1 vote
1 answer
42 views

The behavior of the cross validation error and training error in underfitting case is not clear

I currently study the "Machine Learning" course on Coursera.org by Andrew Ng, it comes to a topic that discusses the performance of learning algorithms under different conditions. Here, we ...
user avatar
4 votes
1 answer
61 views

What could be good Perfomance Evaluation Metrics for a Data Scientist?

Background I'm a Data Scientist and am being asked to come up with a set of metrics/KPIs to assess my annual performance, and things like bonuses (and in the worst case being fired) depend on that. ...
user avatar
0 votes
0 answers
26 views

Techniques to increase the evaluation speed of a neural network

This is somewhat of an open ended question and in some respects a literature request (I would love to be pointed to a survey paper if one exists). Suppose I am constructing a neural network to make ...
user avatar
  • 101
1 vote
0 answers
28 views

How to implement kfold and cv into Hybrid feature selection and evaluate the classification model performance?

I have been working on a Hybrid feature selection combined with hyperopt package for hyperparameter tuning and I am thinking about evaluating the performance of several model classifiers. I looked ...
user avatar
0 votes
1 answer
88 views

Is sensitivity the same as recall in multiclass classification?

In Wikipedia, it is stated "In binary classification, recall is called sensitivity" under the Recall section. Are they both different in case of multi-class classification?
user avatar
0 votes
0 answers
62 views

Why is Pytorch 100x faster than Tensorflow for Deep Reinforcement Implementation?

[Intro] I am currently getting into machine learning coming from mathematics and am following a few courses to get the hang of this. In one of the courses, the author has a github with code examples: ...
user avatar
  • 1
1 vote
1 answer
29 views

How to run hdbscan clustering faster?

I'm using hdbscan to cluster embedding output from BERT, which took in a data file of >150k chat messages. The embedding process took a little over 4 minutes, but as of this writing the hdbscan ...
user avatar
  • 111
0 votes
0 answers
42 views

repeated train/test splitting and assessing performance variability

I have a question related to performance variability and how to assess different methods. I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset). ...
user avatar
  • 321
0 votes
0 answers
24 views

What should be the ratio between training time and accuracy?

What should be the ratio between time and accuracy? I mean when should you drop the accuracy a little bit but it will take less time for the Classifier/Regressor to run? Edit: As part of my studies I ...
user avatar
0 votes
0 answers
8 views

How to incorporate non-present classes when calculating Micro Precision for Non-binary Classification Tasks

I have conducted multiple experiments with my algorithms for non-binary classification. In the most datasets, not all classes in which the algorithm can put the instances are present but only around ...
user avatar
0 votes
0 answers
25 views

How to interpret different coherence values

For an experiment with topic models, I have calculated four coherence values using Gensim's implementation: c_v u_mass c_uci c_npmi From this paper, I know that c_v correlates mostly with human ...
user avatar
  • 189
1 vote
1 answer
18 views

Measure performance of classification model for training on different snapshots

I am trying to do binary classification on some chronological data. Let's assume we have weekly data from the first week of 2017 through the last week of 2020. Now we have found out that 26 weeks of ...
user avatar
  • 179
0 votes
0 answers
27 views

Combine several performance metrics from several datasets

We are developing and evaluating a multi knee/elbow point detection algorithm. For our evaluation, we have 200 sequences of real data. These sequences were annotated manually. For each algorithm and ...
user avatar
1 vote
1 answer
17 views

How to compute performance of a detection-classification system?

I use a yolo (y) to detect only one object and a multiclassifier (mc) that classifies that object. Now, the problem is: what I have to do with yolo's false positive and false negative, if I want to ...
user avatar
-1 votes
1 answer
179 views

nested cross validation vs. train-test split

I am trying to understand the main benefits of conducting a nested cross-validation compared to a simpler train-test split. Let us say I would like to build a prediction model. I initially split my ...
user avatar
  • 1
1 vote
1 answer
20 views

Performance measurement of an event extraction system

I have developed an event extraction system from text documents. It first clusters the data corpus and extracts answers for what, when and where questions. Final answers are determined by using a ...
user avatar
0 votes
1 answer
118 views

What is the Most Efficient Tool in Python for row-wise manipulation of data?

I'm doing a lot of work that requires operations to be performed across rows, using the data in that rows's columns on other columns in the row. I recently had to do some processing on a 1.2 million ...
user avatar
  • 1
2 votes
1 answer
23 views

Create new performance indicators (error metrics)

I am wondering if any of you happen to know of a procedure/approach/rationale to develop new performance indicators (error metrics) that can be used to evaluate the prediction capability (say, ...
user avatar
  • 121
1 vote
0 answers
19 views

correlated variables & model performance: optimal trade-off

on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift. assuming little to medium ...
user avatar
  • 159
1 vote
0 answers
50 views

Creating a new feature from an existing one using decision trees

Is it possible to create a new feature out of two, or more than two existing features using a decision tree? If so, how, and can it produce features with good information value that can better help ...
user avatar
0 votes
0 answers
120 views

2D Z-score/Mahalanobis distance that includes a penalty for uncertainty

I have some 2D points and I want to assess their performance against the target point. When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
user avatar
2 votes
2 answers
113 views

KNN efficient implementation

The KNN algorithm is very handy and particularly suited to some of my problems, but I can't find any resources on how to implement it in production. As a comparative example, when I use a neural ...
user avatar
0 votes
0 answers
32 views

How to increase accuracy and decrease loss of my model

https://jovian.ai/casella0798/badmodel I created the model above to predict red wine quality. I have 6 classes, from 3 to 8. Dataset is unbalanced, with a lot of classes 5 and 6. My model performs ...
user avatar
2 votes
2 answers
46 views

Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set?

I know that choosing between models produced by one algorithm with different hyperparameters the metric for choosing the best one should be the cross-validation on train set. But what about choosing ...
user avatar
  • 157
2 votes
2 answers
50 views

Once a predictive model is in production, how it can be evaluated?

I have a data science project, predicting customer's next purchase day. Customer's one year behavioral data was split to 9 and 3 months for train and test, using RFM analysis, I trained a model with ...
user avatar
  • 23
1 vote
1 answer
772 views

CNN inference is slow on Jetson Nano

I'm running what I believe is a pretty lightweight CNN on an nVidia Jetson Nano with Jetpack 4.4. nVidia claims the Nano can run a ResNet-50 at 36fps, so I expected my much smaller network to run at ...
user avatar
  • 121
1 vote
1 answer
35 views

How do data types influence hardware (CPU / GPU / TPU) performance?

I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
user avatar
  • 2,107
1 vote
1 answer
313 views

What is the appropriate statistical significance test for multi-class classification?

I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have ...
user avatar
3 votes
0 answers
56 views

Fast PR / ROC curves and corespondings AUPR / AUROC

I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC). Its is quite easy to perform those ...
user avatar
  • 2,107
1 vote
0 answers
25 views

How to balance time/effort with transformations, feature selection, and models efficacy in nlp? [closed]

Edit: Question has been edited for reopening (see comment section for justification) Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times ...
user avatar
  • 141
2 votes
1 answer
29 views

Business-related metric for an optimization problem

We have an optimization problem on hand that is the following. Let's say we have 10 different treatments that we might offer, all of them are equally good for us, but people have different propensity ...
user avatar
1 vote
2 answers
122 views

Is fitting two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set more performant than one with 1000?

If I fit two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set, would it have better results than fitting a RandomForestClassifier with 1000 trees and ...
user avatar
3 votes
3 answers
890 views

Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

Suppose I've trained a RandomForest model with 100 trees. I then have two cases: I drop the first tree in the model. I drop the last tree in the model. Would the model performance be less in the ...
user avatar
1 vote
2 answers
68 views

Graphs for representing performance of machine learning classifiers [closed]

Can any please guide about how many types of the graph other than ROC can be plotted to represent the performance of the machine learning classifier?
user avatar
  • 73
4 votes
3 answers
82 views

Finding out why your model is doing better?

I fitted a logistic regression model on a data set and got an AUC score of .70. I added some additional out-hot encoded categorical features to the model and the AUC improved slightly to .74. How do I ...
user avatar
  • 261
0 votes
1 answer
32 views

How to derive false positive and false negative from top-k accuracy?

I am working on the following "equality identification" problem and become quite confused on how to reasonably define false positive and false negative in my case. Problem: Suppose I have a ...
user avatar
1 vote
0 answers
78 views

How to evaluate the performance of a model in production when labeling data is costly?

I have come to a problem for which I can't find a solution. Let's talk about a hypothetical binary classification problem in which you have some years of (human) labeled data. The final objective is ...
user avatar
0 votes
0 answers
17 views

Model performance metrics

I have a dataset with multiple numeric input values and a categorical output. How can I measure model performance with different algorithms. As the results are categorical, we can not obtain r squared ...
user avatar
  • 37
0 votes
2 answers
377 views

Getting lower performance metrics when using GridSearchCV

I have defined an XGBoost model and would like to tune some of its hyperparameters. I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training ...
user avatar
  • 121
0 votes
1 answer
257 views

How to run list comprehensions on GPU?

Is there a way to run complex list comprehensions like the following on GPU? ...
user avatar
2 votes
1 answer
7k views

FP16, FP32 - what is it all about? or is it just Bitsize for Float-Values (Python)

What is it all about FP16, FP32 in Python? My potential Business Partner and I are building a Deep Learning Setup for working with time series. He came up with "FP16 and FP32" while finding a GPU. It ...
user avatar
12 votes
4 answers
9k views

Can the F1 score be equal to zero?

As it is mentioned in the F1 score Wikipedia, 'F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0'. What is the worst condition that was mentioned? Even if we ...
user avatar
1 vote
1 answer
1k views

LSTM evaluation metric MAE explanation

I have a hard time understanding the LSTM model performance as I summarize my model as follow: ...
user avatar
  • 111