Questions tagged [accuracy]

In data science, accuracy is a measurement used to determine which model is best at describing the underlying patterns of a dataset.

Filter by
Sorted by
Tagged with
92
votes
4answers
92k views

Advantages of AUC vs standard accuracy

I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've ...
43
votes
4answers
19k views

Is it always better to use the whole dataset to train the final model?

A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
36
votes
8answers
7k views

What would I prefer - an over-fitted model or a less accurate model?

Let's say we have two models trained. And let's say we are looking for good accuracy. The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted. The second has an ...
25
votes
1answer
21k views

What is a LB score in machine learning?

I was going through an article on kaggle blogs. Repeatedly, the author mentions 'LB score' and 'LB fit') as a metric for effectiveness of machine learning (along with cross validation (CV) score). ...
23
votes
3answers
501 views

How do you manage expectations at work?

With all the hoopla around Data Science, Machine Learning, and all the success stories around, there are a lot of both justified, as well as overinflated, expectations from Data Scientists and their ...
20
votes
2answers
11k views

How to increase accuracy of classifiers?

I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and ...
16
votes
1answer
59k views

Train Accuracy vs Test Accuracy vs Confusion matrix

After I developed my predictive model using Random Forest I get the following metrics: ...
15
votes
3answers
28k views

macro average and weighted average meaning in classification_report

I use the "classification_report" from from sklearn.metrics import classification_report in order to evaluate the imbalanced binary classification ...
11
votes
2answers
14k views

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

I have read on the several answers here and on the Internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting. But I am confused that which ...
11
votes
3answers
5k views

Inverse Relationship Between Precision and Recall

I made some search to learn precision and recall and I saw some graphs represents inverse relationship between precision and recall and I started to think about it to clarify subject. I wonder the ...
11
votes
3answers
9k views

Relationship between KS, AUROC, and Gini

Common model validation statistics like the Kolmogorov–Smirnov test (KS), AUROC, and Gini coefficient are all functionally related. However, my question has to do with proving how these are all ...
10
votes
2answers
5k views

How to get an aggregate confusion matrix from n different classifications

I want to test the accuracy of a methodology. I ran it ~400 times, and I got a different classification for each run. I also have the ground truth, i.e., the real classification to test against. For ...
9
votes
5answers
13k views

In which epoch should i stop the training to avoid overfitting

I'm working on an age estimation project trying to classify a given face in a predefined age range. For that purpose I'm training a deep NN using the keras library. The accuracy for the training and ...
9
votes
7answers
29k views

I got 100% accuracy on my test set,is there something wrong?

I got 100% accuracy on my test set when trained using decision tree algorithm.but only got 85% accuracy on random forest Is there something wrong with my model or is decision tree best suited for the ...
8
votes
1answer
31k views

How to know if a model is overfitting or underfitting by looking at graph

Just recently got my hands on tensorboard, but can you tell me what features should I look for in the graph (Accuracy and Validation Accuracy) And please do enlighten me about the concept of ...
8
votes
1answer
3k views

Balanced Accuracy vs. F1 Score

I've read plenty of online posts with clear explanations about the difference between accuracy and F1 score in a binary classification context. However, when I came across the concept of balanced ...
7
votes
3answers
2k views

0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial

My goal is to classify products pictures into categories such as dress, sandals, etc. I am using the MNIST fashion dataset, following this official tutorial word-per-word: https://www.tensorflow.org/...
7
votes
2answers
195 views

What makes you confident in your results? At what point do you think you can present your work to tech illiterate superiors?

I understand that the models are only as good as the data you get, and bad design can generate really bad data. Nonrandom sampling, unbalanced/incomplete designs, confounding, can make data analysis ...
7
votes
2answers
734 views

Coursera ML - Does the choice of optimization algorithm affect the accuracy of multiclass logistic regression?

I recently completed exercise 3 of Andrew Ng's Machine Learning on Coursera using Python. When initially completing parts 1.4 to 1.4.1 of the exercise, I ran into difficulties ensuring that my ...
7
votes
4answers
5k views

Log loss vs accuracy for deciding between different learning rates?

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately. Log loss When I used log loss as score in ...
7
votes
2answers
218 views

Why isn't dimension sampling used with gradient boosting machines (GBM)?

GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used ...
6
votes
4answers
20k views

How to improve accuracy of deep neural networks

I am using Tensorflow to predict whether the given sentence is positive and negative. I have take 5000 samples of positive sentences and 5000 samples of negative sentences. 90% of the data I used it ...
6
votes
3answers
7k views

How to determine if my GBM model is overfitting?

Below is a simplified example of a h2o gradient boosting machine model using R's iris dataset. The model is trained to predict sepal length. The example yields an r2 value of 0.93, which seems ...
6
votes
4answers
18k views

How to improve loss and avoid overfitting

I'm trying to build a 2 class image classifier using the architecture suggested in first part of this blog https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data....
6
votes
2answers
2k views

Why validation loss worsens while precision/recall continue to improve?

I'm training a neural network on 'easy' dataset with ~15k examples. Network overfits pretty fast. The thing I cannot understand that after 5th epoch validation loss is starting to worsen, while ...
6
votes
1answer
97 views

How to define a custom resampling methodology

I'm using an experimental design to test the robustness of different classification methods, and now I'm searching for the correct definition of such design. I'm creating different subsets of the ...
5
votes
2answers
280 views

scikit-learn RandomForestClassifier always hits 100% test accuracy

I have been playing with a toy problem to compare the performance and behavior of several scikit-learn classifiers. Brief, I have one continuous variable X (which contains two samples of size N, each ...
5
votes
4answers
15k views

How to compute f1 in TensorFlow

I have a code that computes the accuracy, but now I would like to compute the F1 score. ...
5
votes
3answers
1k views

Accuracy is lower than f1-score for imbalanced data

For a binary classification, I have a dataset with 55% negative label and 45% positive labels. The results of the classifier shows that the accuracy is lower than the f1-score. Does that mean that the ...
5
votes
1answer
3k views

What is the classification accuracy of a random classifier?

I have a build a classification model using machine learning technique (SVM). I want to compare the classification accuracy of my model with a random classifier. My data set contains only two classes(...
5
votes
1answer
2k views

Training and cross validation error curves

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve. It is observed that the accuracy of training dataset decreases but ...
5
votes
1answer
966 views

Multiclass classification on imbalanced dataset : Accuracy or micro F1 or macro F1

I have a multiclass classification problem. Further, an instance can be assigned to exactly one class. My dataset is highly imbalanced. I know that accuracy is not a good metric to use in this case ...
5
votes
2answers
3k views

Classification Accuracy in Keras

I'm using two different functions to calculate the accuracy of my deep learning model and I am confused which one is which. The first one is ...
5
votes
1answer
669 views

Why does degradation occur in deep neural networks?

It has been shown that "plain" neural networks tend to have an increased amount training error, and accompanied test error, as more layers are added. I am not quite certain as to why this occurs. In ...
4
votes
2answers
3k views

human level performance on ImageNet, top-1 or top-5?

Anyone have pointers to where the human level performance on ImageNet comes from? I found a reference to 5.1% accuracy (top-1? or top-5?) from here.
4
votes
2answers
752 views

Why are results without Transfer Learning better than with Transfer Learning?

I developed a neural network for license plate recognition and used the EfficientNet architecture (https://keras.io/api/applications/efficientnet/#efficientnetb0-function) with and without pretrained ...
4
votes
1answer
599 views

What does a predicted probability really mean, without considering the accuracy of the underlying model?

Say I've built a (completely unrealistic) classification model in Keras that gives me 1.00 accuracy. And next, I would like to use my model on some new, unseen data, and use ...
4
votes
1answer
392 views

Performance of model in production varying greatly from train-test data

I was wondering if anyone has any advice on where to start digging for this problem. I have a model which has gone through development and all train/cv/test data sets now perform above 95% both for ...
4
votes
3answers
130 views

Is my model over fitting or not?

I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I ...
3
votes
2answers
3k views

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Logistic regression is supposed to work well only on data that is linearly separable. As we can see in the pair plot, the data points heavily overlap. The logistic regression model is in fact showing ...
3
votes
4answers
1k views

Why is my test data accuracy higher than my training data?

I'm using four years of data, training on the first 3 and testing on the fourth. Using LSTM w/ Keras. My test data set (which has no overlap at all with the training) is consistently performing better ...
3
votes
2answers
363 views

How to compare paired count data?

I am working with a machine learning approach that counts cars in images. I have a predicted dataset, which is the predicted output from the machine learning approach and a paired "true" dataset, ...
3
votes
1answer
6k views

99% validation accuracy but 0% prediction results (UNET Architecture)

I am debugging results from the UNET architecture that I am using for identifying corneal reflection in eye images. While I am getting over 99% training accuracy and also very high (over 99%) ...
3
votes
3answers
9k views

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

I am evaluating a Credit Risk model that predicts the estimated likelihood of customers defaulting on their mortgage accounts. The model is a Logistic Regression estimator and was built by another ...
3
votes
3answers
489 views

Check Accuracy of Model Provided by Consultant

My company has recently engaged a consultant firm to develop a predictive model to detect defective works. I understand that there are many ways to validate the model, for example, using k-fold cross-...
3
votes
4answers
1k views

Metrics to determine K in K-cross fold validation

Consider a scenario where the dataset in hand is quite large, let's assume 50000 samples (quite well balanced between two classes). What metrics can be used to decide the K value in a K-fold cross-...
3
votes
2answers
51 views

Good approach to increase accuracy for a continuous value that is highly variable/sensitive to the inputs?

I am trying to predict a continuous 'Y' variable using a variety of algorithms and feature engineering techniques. My issue is that Y is extremely variable and I reached a asymptote in accuracy. This ...
3
votes
2answers
2k views

Calculate average Intersection over Union

I want to have a global IoU metric for each class in a segmentation model with a neural net. The idea is, once the net is trained, doing the forward pass over all training examples an calculate the ...
3
votes
1answer
241 views

Avoid iterations while calculating average model accuracy

I am fitting a model in R. use createFolds method to create several k folds from the data set loop through the folds, repeating ...
3
votes
2answers
318 views

Making sense of loss and accuracy curves

This is an issue that I have come across over and over again. Loss (cross-entropy in this case) and accuracy plots that do not make sense. Here is an example: Here, I’m training a ReNet18 on CIFAR10. ...

1
2 3 4 5
7