# Questions tagged [scoring]

The tag has no usage guidance.

58 questions
Filter by
Sorted by
Tagged with
23 views

### Constructing a predictive scoring system for a binary output

I have a system that has several continuous variables as inputs. The output is a binary pass/fail type. The relationship between the continuous variables and the changes of a "pass" is that ...
1 vote
95 views

### weighting voting classifier (MAE and MSE)

I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula: score = w * MAE + (w-1) * ...
94 views

### Representation of Strictly Proper Scoring Rule for Multiclasss Classificaiton

I am working on a classification problem, using features $\mathbf{x}$ to predict a target variable $y \in \mathbb{N}_0$. By a strictly proper scoring rule I mean a loss function $\ell(y,\hat{y})$ for ...
• 103
76 views

### Likelihood function of a Beta-Normal distribution

I'm going over a paper regarding the calculation of credit scoring By Kalkbrener and Onwunta. In the article, they derive the following Likelihood function, in order to find the MLE for $R^2$: (...
• 101
82 views

### Why does my custom rmsle_loss produce negative scores during cross-validation?

I have a pipeline and scorer that produces some unexpected behaviors during cross-validation. During cross validation the pipeline produces all negative scores, but from the definition from rmsle_loss ...
• 5
19 views

### Help creating a SCORING model/formula.......am I on the right track?

Trying to create a basic machine scoring model, that takes in 4 parameters: Number of maintenance events Years of life left Manufacturer support (bit - either yes or no) Visual condition The ...
1 vote
49 views

### Is there a way to quantify uncertainty in classification?

I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that ...
• 65
49 views

### Scoring unsupervised data

Given the features and details of a house listed for selling, i'd wish to calculate a score based on the accuracy, completeness, genuineness and authenticity provided by the owner.According to me, ...
1 vote
191 views

### Ways of scaling scores on data without knowing possible maximum values

In my scenario, I have to process some input data and give a score based on what the processing phase outputs. The problem is that, in order to scale the score in a human-readable format I'd have to ...
241 views

### Approaches for matching leads to salesmen

I'm starting to tackle a new problem where we are trying to optimally match new leads (perspective customers) for our product to our sales representatives in the hopes of improving bottom-line metrics ...
28 views

### How can i adapt accuracy metric for multiclass classification?

I have a problem which is multiclass e.g. That is 4 classes. I would like a custom metric to assess the model where only if class 3 is predicted as class 2 and class 2 is predicted as class 3 (i.e. ...
• 526
1 vote
26 views

### Right way to compare model scores for Next Best Action

I have around 15 classification models for different products built in different ways (some are RF, some are Gradient Boosting, some were downsampled in one way, others in other way, some are built in ...
1 vote
891 views

### Scikit-learn make_scorer custom metric problem for multiclass clasification

I was doing a churn analysis using: ...
43 views

### Data science tools for easing the participation of a business into their scoring system

I'm a working in a small company. The company sells products on a website and they have a python script that runs everyday to attribute a score to each product based on a set of parameters (google ...
1 vote
13 views

### Scoring samples after clusterings [closed]

I want to assign a score to all points in a group that I cluster several time. I want the score to indicate how much this point is grouped with the same individuals all time. I suppose this idea ...
• 323
617 views

### Having trouble scaling scores of logistic regression

I am constructing a credit scorecard using logistic regression, similar to the one shown here. However, when trying to convert the coefficients of logistic regression into score representation (by ...
• 225
486 views

### What is the proper way to bin variables for calculating WoE during credit scoring?

I have read this article about developing a credit scorecard in python, where it is stated that when binning the continuous variables, it needs to be ensured that: ...
• 225
1 vote
39 views

### How to score the health of a company? [closed]

i'm currently doing dual apprenticeships. My main mission is to represent the health of a company based on accounting records for multiple companies over multiple years. The part of an accounting ...
1 vote
142 views

### What Machine Learning Technique can I use to judge boxing fights? [closed]

I want to build a machine learning model that judges the fights based on the results of each round. Any suggestions on what techniques can I use?
• 111
1 vote
179 views

### xgboost calibration kde plots (isotonic) not smooth

i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the ...
• 526
1 vote
841 views

### What is the best way to calculate score based on input features?

Lets say we have certain products. We also have certain input features regarding these products like inventory, sales, price, cost etc. based on these features we want to score these parts. The score ...
• 111
292 views

### How to do class balancing?

I am working with a really imbalanced dataset ($\approx$ 1% of positive cases) for a classification problem. I know that class balancing is an important step in this scenario. I have two questions: ...
• 193
24 views

### Weighted Average Score

I have several different scores that I'd like to combine them into a single score that represents all parameters. Score components are: Nutrition Score (1-10): 1 for the worst quality and 10 for the ...
1 vote
2k views

### How to compute AUC in gridsearchSV (multiclass problem)

I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the ...
• 13
1 vote
88 views

### scaling credit risk scorecard

I need to build a credit risk scorecard using logistic and linear regression. The variables using to predict are all dummies, where each dummy is a bin of some variable. Let's say the variable age, I ...
83 views

### Random forest mode scoring

We are using random forest algorithm but having some trouble understanding the scoring method it uses. take for example the following CM of the test set: ...
• 201
44 views

### Scoring Methodology

Consider a the below provided sample data; ...
• 65
1 vote
73 views

### Scoring products across all stores

I am trying to score each product based off a couple of features. I was thinking I can use the sales rate(7 days prior) as the target column and predict the sales rate using some ensemble technique ...
127 views

### Best practices for scoring hundreds of models on the same massive dataset?

I have 500+ models predicting various things and a massive database of over 400m+ individuals and about 5,000 possible independent variables. Currently, my scoring process takes about 5 days, and ...
• 215
1 vote
2k views

### How to create a score for a SWOT analysis (strengths, weaknesses, opportunities, and threats)?

I'm developing a participatory social environmental diagnostic. To do this, I'm using primary (qualitative data from interviews with stakeholders) and secondary data (local socioeconomic data). From ...
• 113
1 vote
4k views

### Which scoring for GridSearchCV is best, when imbalanced multiclass dataset?

I have an unbalanced multiclass dataset (GTSRB) and want to optimize the hyperparameters of an SVM through GridSearchCV. I know that accuracy is not suitable for scoring in this case. Which evaluation ...
• 403
869 views

### sklearn.metrics.average_precision_score getting different answers for same data but different formats

I was trying to learn how average precision (AP) is calculated and implemented in scikit-learn. I have read the documentation, but I don't think I fully understand it yet. Consider the following two ...
• 33
96 views

### what approach to use for find best customer out of data?

I'm working on this project where the objective is to find certain good leads/customers from the existing customer dataset. I tried the RFM method for scoring but there is no data regarding money or ...
1 vote
17 views

### Is there a term for measuring error on a second prediction based on the first's?

I have created a dataset which contains six values per row which may be the target value. Two rows for example: ...
• 111
3k views

There is a example in sklearn.metrics.average_precision_score documentation. ...
1 vote
19 views

### Assessing performance of an agent based on commission rate, market share and revenue

I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following: Number ...
• 85
93 views

### Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
• 65
11k views

### How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn

I'm trying to get mean test scores from scikit-learn's GridSearchCV with multiple scorers. grid.cv_results_ displays lots of ...
• 410
634 views

### How to interpret Sum of Squared Error in a classification task

I am working on ANN. I have 2497 training examples and each of them is a vector of 128, so the input size is 128. Number of neurons in hidden layer is 64 and number of output neurons is 6 (since ...
4k views

### Credit scoring using scorecardpy with XGBoost

I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried ...
• 51
311 views

### Is there a definitive and more conclusive way of interpreting the R^2 score from a linear regression model in terms of prediction accuracy?

I'm trying to find a definitive way to conclude the R^2 score from a prediction accuracy point of view rather than variance. How should I do it? Conceptually, most blogs / articles explain R^2 as: ...
• 31
2k views

### Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
• 3,548
274 views

### How can I compare classes from clusterings performed on two different data sets?

I have two data sets defined by real valued vectors, and I have performed clustering on both of them. Now I want to compare the classes to see how they map to each other. If I put the data sets ...
• 143
1 vote
99 views

### Predict class having only class proportions for every attribute (non labeled data)

I am working with a big data set (millions of observations) where for each observation I am trying to predict a probability (or score it) of being of a class. I haven't any labeled training data and ...
43 views

### How to approach model reporting task

I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates ...
• 254
73 views

### Is the maximum BDeu Bayesian Network always the empty network?

I'm recently reading a paper about Scoring Mechanisms for Bayesian Networks. For the BDeu score, it appears that the maximum possible score of BDeu for Bayesian Network structure learning is zero. ...
689 views

### Ranking algorithm based on a handful of features

I am trying to determine the apt algorithm for a ranking problem that I am working on. I have social media metrics - engagement, sentiment, audience size etc for several brands and am looking for a ...
• 310
1 vote
82 views

### What is an appropriate way to compare classifiers with different sets of classes?

I have three classifiers for language identification: A: en, de, ru, fr, ij, kl B: en, de, ru, fr, xy C: en, de, ru, fr, no, pq, rs and I have a balanced dataset ...
• 18.9k
46 views

### Intuitive interpretation of ratios between training set scores and validation set scores

I'm training models with the usual setup where you hold back a portion (in my case, 20%) of the data just to see how your trained model generalizes to unseen data, to see if it's overfitting. When ...
• 211