Questions tagged [scoring]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

Constructing a predictive scoring system for a binary output

I have a system that has several continuous variables as inputs. The output is a binary pass/fail type. The relationship between the continuous variables and the changes of a "pass" is that ...
Cusco88's user avatar
1 vote
2 answers
71 views

weighting voting classifier (MAE and MSE)

I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula: score = w * MAE + (w-1) * ...
Guilherme Raibolt's user avatar
3 votes
1 answer
91 views

Representation of Strictly Proper Scoring Rule for Multiclasss Classificaiton

I am working on a classification problem, using features $\mathbf{x}$ to predict a target variable $y \in \mathbb{N}_0$. By a strictly proper scoring rule I mean a loss function $\ell(y,\hat{y})$ for ...
user1337's user avatar
  • 103
0 votes
1 answer
68 views

Likelihood function of a Beta-Normal distribution

I'm going over a paper regarding the calculation of credit scoring By Kalkbrener and Onwunta. In the article, they derive the following Likelihood function, in order to find the MLE for $R^2$: (...
Alex Teush's user avatar
0 votes
1 answer
59 views

Why does my custom rmsle_loss produce negative scores during cross-validation?

I have a pipeline and scorer that produces some unexpected behaviors during cross-validation. During cross validation the pipeline produces all negative scores, but from the definition from rmsle_loss ...
Tim's user avatar
  • 5
0 votes
0 answers
19 views

Help creating a SCORING model/formula.......am I on the right track?

Trying to create a basic machine scoring model, that takes in 4 parameters: Number of maintenance events Years of life left Manufacturer support (bit - either yes or no) Visual condition The ...
manavjn's user avatar
0 votes
0 answers
57 views

Why does my accuracy score drop after hyperparameter tuning in XGBoost?

I am trying to tune the model I've built, but every time I change hyperparameters my accuracy score drops significantly. I'm using RandomizedSearchCV and best_params_ to determine which parameters I ...
mvinegret's user avatar
0 votes
0 answers
10 views

What are some commonly used non-threshold dependent classification predictive performance metrics?

I'm working on a prediction framework that utilizes a classification algorithm, trained on data with a binary outcome, and makes probabilistic predictions. I'm looking for some help determining ...
Ted's user avatar
  • 1
1 vote
0 answers
44 views

Is there a way to quantify uncertainty in classification?

I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that ...
Tom's user avatar
  • 65
0 votes
1 answer
48 views

Scoring unsupervised data

Given the features and details of a house listed for selling, i'd wish to calculate a score based on the accuracy, completeness, genuineness and authenticity provided by the owner.According to me, ...
Abhijeet varun's user avatar
1 vote
2 answers
174 views

Ways of scaling scores on data without knowing possible maximum values

In my scenario, I have to process some input data and give a score based on what the processing phase outputs. The problem is that, in order to scale the score in a human-readable format I'd have to ...
Ionut-Alexandru Baltariu's user avatar
0 votes
1 answer
210 views

Approaches for matching leads to salesmen

I'm starting to tackle a new problem where we are trying to optimally match new leads (perspective customers) for our product to our sales representatives in the hopes of improving bottom-line metrics ...
bkubs55577's user avatar
0 votes
0 answers
28 views

How can i adapt accuracy metric for multiclass classification?

I have a problem which is multiclass e.g. That is 4 classes. I would like a custom metric to assess the model where only if class 3 is predicted as class 2 and class 2 is predicted as class 3 (i.e. ...
Maths12's user avatar
  • 516
1 vote
0 answers
26 views

Right way to compare model scores for Next Best Action

I have around 15 classification models for different products built in different ways (some are RF, some are Gradient Boosting, some were downsampled in one way, others in other way, some are built in ...
user8419142's user avatar
1 vote
1 answer
843 views

Scikit-learn make_scorer custom metric problem for multiclass clasification

I was doing a churn analysis using: ...
Antonio Velazquez Bustamante's user avatar
0 votes
1 answer
43 views

Data science tools for easing the participation of a business into their scoring system

I'm a working in a small company. The company sells products on a website and they have a python script that runs everyday to attribute a score to each product based on a set of parameters (google ...
Matthieu Dsprz's user avatar
1 vote
0 answers
13 views

Scoring samples after clusterings [closed]

I want to assign a score to all points in a group that I cluster several time. I want the score to indicate how much this point is grouped with the same individuals all time. I suppose this idea ...
EzrielS's user avatar
  • 323
2 votes
1 answer
557 views

Having trouble scaling scores of logistic regression

I am constructing a credit scorecard using logistic regression, similar to the one shown here. However, when trying to convert the coefficients of logistic regression into score representation (by ...
Ach113's user avatar
  • 225
0 votes
1 answer
465 views

What is the proper way to bin variables for calculating WoE during credit scoring?

I have read this article about developing a credit scorecard in python, where it is stated that when binning the continuous variables, it needs to be ensured that: ...
Ach113's user avatar
  • 225
1 vote
1 answer
39 views

How to score the health of a company? [closed]

i'm currently doing dual apprenticeships. My main mission is to represent the health of a company based on accounting records for multiple companies over multiple years. The part of an accounting ...
Vivien Leonard's user avatar
1 vote
1 answer
142 views

What Machine Learning Technique can I use to judge boxing fights? [closed]

I want to build a machine learning model that judges the fights based on the results of each round. Any suggestions on what techniques can I use?
bisamov's user avatar
  • 111
1 vote
0 answers
174 views

xgboost calibration kde plots (isotonic) not smooth

i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the ...
Maths12's user avatar
  • 516
1 vote
2 answers
799 views

What is the best way to calculate score based on input features?

Lets say we have certain products. We also have certain input features regarding these products like inventory, sales, price, cost etc. based on these features we want to score these parts. The score ...
Max08's user avatar
  • 111
3 votes
3 answers
262 views

How to do class balancing?

I am working with a really imbalanced dataset ($\approx$ 1% of positive cases) for a classification problem. I know that class balancing is an important step in this scenario. I have two questions: ...
A1010's user avatar
  • 193
0 votes
0 answers
23 views

Weighted Average Score

I have several different scores that I'd like to combine them into a single score that represents all parameters. Score components are: Nutrition Score (1-10): 1 for the worst quality and 10 for the ...
Ars T.'s user avatar
  • 9
1 vote
3 answers
2k views

How to compute AUC in gridsearchSV (multiclass problem)

I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the ...
okraw's user avatar
  • 13
1 vote
0 answers
86 views

scaling credit risk scorecard

I need to build a credit risk scorecard using logistic and linear regression. The variables using to predict are all dummies, where each dummy is a bin of some variable. Let's say the variable age, I ...
user12195705's user avatar
2 votes
1 answer
80 views

Random forest mode scoring

We are using random forest algorithm but having some trouble understanding the scoring method it uses. take for example the following CM of the test set: ...
Amit Raz's user avatar
  • 201
0 votes
1 answer
44 views

Scoring Methodology

Consider a the below provided sample data; ...
James's user avatar
  • 65
1 vote
0 answers
72 views

Scoring products across all stores

I am trying to score each product based off a couple of features. I was thinking I can use the sales rate(7 days prior) as the target column and predict the sales rate using some ensemble technique ...
Vidya Anandamurali's user avatar
2 votes
3 answers
124 views

Best practices for scoring hundreds of models on the same massive dataset?

I have 500+ models predicting various things and a massive database of over 400m+ individuals and about 5,000 possible independent variables. Currently, my scoring process takes about 5 days, and ...
boot-scootin's user avatar
1 vote
1 answer
2k views

How to create a score for a SWOT analysis (strengths, weaknesses, opportunities, and threats)?

I'm developing a participatory social environmental diagnostic. To do this, I'm using primary (qualitative data from interviews with stakeholders) and secondary data (local socioeconomic data). From ...
Britto's user avatar
  • 113
1 vote
2 answers
4k views

Which scoring for GridSearchCV is best, when imbalanced multiclass dataset?

I have an unbalanced multiclass dataset (GTSRB) and want to optimize the hyperparameters of an SVM through GridSearchCV. I know that accuracy is not suitable for scoring in this case. Which evaluation ...
Code Now's user avatar
  • 393
3 votes
1 answer
835 views

sklearn.metrics.average_precision_score getting different answers for same data but different formats

I was trying to learn how average precision (AP) is calculated and implemented in scikit-learn. I have read the documentation, but I don't think I fully understand it yet. Consider the following two ...
ychen's user avatar
  • 33
2 votes
1 answer
91 views

what approach to use for find best customer out of data?

I'm working on this project where the objective is to find certain good leads/customers from the existing customer dataset. I tried the RFM method for scoring but there is no data regarding money or ...
Akshit jain's user avatar
1 vote
0 answers
17 views

Is there a term for measuring error on a second prediction based on the first's?

I have created a dataset which contains six values per row which may be the target value. Two rows for example: ...
CBusBus's user avatar
  • 111
2 votes
1 answer
3k views

About sklearn.metrics.average_precision_score documentation

There is a example in sklearn.metrics.average_precision_score documentation. ...
disney82231's user avatar
1 vote
0 answers
19 views

Assessing performance of an agent based on commission rate, market share and revenue

I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following: Number ...
HaneenSu's user avatar
2 votes
1 answer
91 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
James's user avatar
  • 65
4 votes
1 answer
11k views

How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn

I'm trying to get mean test scores from scikit-learn's GridSearchCV with multiple scorers. grid.cv_results_ displays lots of ...
jeffhale's user avatar
  • 400
2 votes
1 answer
583 views

How to interpret Sum of Squared Error in a classification task

I am working on ANN. I have 2497 training examples and each of them is a vector of 128, so the input size is 128. Number of neurons in hidden layer is 64 and number of output neurons is 6 (since ...
user60080's user avatar
4 votes
1 answer
4k views

Credit scoring using scorecardpy with XGBoost

I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried ...
Minila S's user avatar
3 votes
2 answers
285 views

Is there a definitive and more conclusive way of interpreting the R^2 score from a linear regression model in terms of prediction accuracy?

I'm trying to find a definitive way to conclude the R^2 score from a prediction accuracy point of view rather than variance. How should I do it? Conceptually, most blogs / articles explain R^2 as: ...
z-wei's user avatar
  • 31
13 votes
3 answers
2k views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
Bruno Lubascher's user avatar
4 votes
2 answers
267 views

How can I compare classes from clusterings performed on two different data sets?

I have two data sets defined by real valued vectors, and I have performed clustering on both of them. Now I want to compare the classes to see how they map to each other. If I put the data sets ...
Okarin's user avatar
  • 143
1 vote
0 answers
98 views

Predict class having only class proportions for every attribute (non labeled data)

I am working with a big data set (millions of observations) where for each observation I am trying to predict a probability (or score it) of being of a class. I haven't any labeled training data and ...
Artem Nagornui's user avatar
2 votes
1 answer
41 views

How to approach model reporting task

I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates ...
Sledge's user avatar
  • 254
3 votes
1 answer
70 views

Is the maximum BDeu Bayesian Network always the empty network?

I'm recently reading a paper about Scoring Mechanisms for Bayesian Networks. For the BDeu score, it appears that the maximum possible score of BDeu for Bayesian Network structure learning is zero. ...
jamesmiraflor's user avatar
3 votes
1 answer
679 views

Ranking algorithm based on a handful of features

I am trying to determine the apt algorithm for a ranking problem that I am working on. I have social media metrics - engagement, sentiment, audience size etc for several brands and am looking for a ...
kms's user avatar
  • 310
1 vote
0 answers
81 views

What is an appropriate way to compare classifiers with different sets of classes?

I have three classifiers for language identification: A: en, de, ru, fr, ij, kl B: en, de, ru, fr, xy C: en, de, ru, fr, no, pq, rs and I have a balanced dataset ...
Martin Thoma's user avatar
  • 18.9k