Questions tagged [scoring]
The scoring tag has no usage guidance.
61
questions
1
vote
2
answers
60
views
weighting voting classifier (MAE and MSE)
I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula:
score = w * MAE + (w-1) * ...
2
votes
1
answer
86
views
Representation of Strictly Proper Scoring Rule for Multiclasss Classificaiton
I am working on a classification problem, using features $\mathbf{x}$ to predict a target variable $y \in \mathbb{N}_0$. By a strictly proper scoring rule I mean a loss function $\ell(y,\hat{y})$ for ...
- 93
0
votes
1
answer
58
views
Likelihood function of a Beta-Normal distribution
I'm going over a paper regarding the calculation of credit scoring By Kalkbrener and Onwunta. In the article, they derive the following Likelihood function, in order to find the MLE for $R^2$:
(...
- 101
0
votes
1
answer
41
views
Why does my custom rmsle_loss produce negative scores during cross-validation?
I have a pipeline and scorer that produces some unexpected behaviors during cross-validation. During cross validation the pipeline produces all negative scores, but from the definition from rmsle_loss ...
- 5
0
votes
0
answers
16
views
Help creating a SCORING model/formula.......am I on the right track?
Trying to create a basic machine scoring model, that takes in 4 parameters:
Number of maintenance events
Years of life left
Manufacturer support (bit - either yes or no)
Visual condition
The ...
- 1
0
votes
0
answers
48
views
Why does my accuracy score drop after hyperparameter tuning in XGBoost?
I am trying to tune the model I've built, but every time I change hyperparameters my accuracy score drops significantly. I'm using RandomizedSearchCV and best_params_ to determine which parameters I ...
0
votes
0
answers
8
views
What are some commonly used non-threshold dependent classification predictive performance metrics?
I'm working on a prediction framework that utilizes a classification algorithm, trained on data with a binary outcome, and makes probabilistic predictions. I'm looking for some help determining ...
- 1
0
votes
0
answers
11
views
Assigning Final Scores to Identified Technologies: Considering Users' Reputation, badge counts, post scores, no.of posts, & post date
I am trying to determine the importance of various factors in assigning a score for identified technologies using the user's StackOverflow post tags and content.
The considering factors are users' ...
- 103
1
vote
0
answers
40
views
Is there a way to quantify uncertainty in classification?
I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that ...
- 63
0
votes
0
answers
42
views
Reject Inference: proportion between rejects and accepts
I am trying to do a project on developing a credit scorecard with reject inference.
The probem is that the instances have a low approval rate, i.e. a small targeted population and a very big rejected ...
0
votes
1
answer
38
views
Scoring unsupervised data
Given the features and details of a house listed for selling, i'd wish to calculate a score based on the accuracy, completeness, genuineness and authenticity provided by the owner.According to me, ...
1
vote
2
answers
139
views
Ways of scaling scores on data without knowing possible maximum values
In my scenario, I have to process some input data and give a score based on what the processing phase outputs. The problem is that, in order to scale the score in a human-readable format I'd have to ...
0
votes
1
answer
165
views
Approaches for matching leads to salesmen
I'm starting to tackle a new problem where we are trying to optimally match new leads (perspective customers) for our product to our sales representatives in the hopes of improving bottom-line metrics ...
0
votes
0
answers
28
views
How can i adapt accuracy metric for multiclass classification?
I have a problem which is multiclass e.g. That is 4 classes. I would like a custom metric to assess the model where only if class 3 is predicted as class 2 and class 2 is predicted as class 3 (i.e. ...
- 506
1
vote
0
answers
26
views
Right way to compare model scores for Next Best Action
I have around 15 classification models for different products built in different ways (some are RF, some are Gradient Boosting, some were downsampled in one way, others in other way, some are built in ...
- 11
1
vote
1
answer
795
views
Scikit-learn make_scorer custom metric problem for multiclass clasification
I was doing a churn analysis using:
...
0
votes
1
answer
42
views
Data science tools for easing the participation of a business into their scoring system
I'm a working in a small company. The company sells products on a website and they have a python script that runs everyday to attribute a score to each product based on a set of parameters (google ...
- 103
1
vote
0
answers
13
views
Scoring samples after clusterings [closed]
I want to assign a score to all points in a group that I cluster several time.
I want the score to indicate how much this point is grouped with the same individuals all time.
I suppose this idea ...
- 323
2
votes
1
answer
483
views
Having trouble scaling scores of logistic regression
I am constructing a credit scorecard using logistic regression, similar to the one shown here. However, when trying to convert the coefficients of logistic regression into score representation (by ...
- 225
0
votes
1
answer
421
views
What is the proper way to bin variables for calculating WoE during credit scoring?
I have read this article about developing a credit scorecard in python, where it is stated that when binning the continuous variables, it needs to be ensured that:
...
- 225
1
vote
1
answer
39
views
How to score the health of a company? [closed]
i'm currently doing dual apprenticeships. My main mission is to represent the health of a company based on accounting records for multiple companies over multiple years. The part of an accounting ...
1
vote
1
answer
136
views
What Machine Learning Technique can I use to judge boxing fights? [closed]
I want to build a machine learning model that judges the fights based on the results of each round. Any suggestions on what techniques can I use?
- 111
1
vote
0
answers
165
views
xgboost calibration kde plots (isotonic) not smooth
i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the ...
- 506
1
vote
2
answers
720
views
What is the best way to calculate score based on input features?
Lets say we have certain products. We also have certain input features regarding these products like inventory, sales, price, cost etc. based on these features we want to score these parts. The score ...
- 111
3
votes
3
answers
247
views
How to do class balancing?
I am working with a really imbalanced dataset ($\approx$ 1% of positive cases) for a classification problem. I know that class balancing is an important step in this scenario.
I have two questions:
...
- 193
0
votes
0
answers
22
views
Weighted Average Score
I have several different scores that I'd like to combine them into a single score that represents all parameters. Score components are:
Nutrition Score (1-10): 1 for the worst quality and 10 for the ...
- 9
1
vote
3
answers
1k
views
How to compute AUC in gridsearchSV (multiclass problem)
I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the ...
- 13
1
vote
0
answers
79
views
scaling credit risk scorecard
I need to build a credit risk scorecard using logistic and linear regression. The variables using to predict are all dummies, where each dummy is a bin of some variable.
Let's say the variable age, I ...
- 11
2
votes
1
answer
77
views
Random forest mode scoring
We are using random forest algorithm but having some trouble understanding the scoring method it uses.
take for example the following CM of the test set:
...
- 201
0
votes
1
answer
39
views
1
vote
0
answers
71
views
Scoring products across all stores
I am trying to score each product based off a couple of features. I was thinking I can use the sales rate(7 days prior) as the target column and predict the sales rate using some ensemble technique ...
2
votes
3
answers
114
views
Best practices for scoring hundreds of models on the same massive dataset?
I have 500+ models predicting various things and a massive database of over 400m+ individuals and about 5,000 possible independent variables.
Currently, my scoring process takes about 5 days, and ...
- 215
1
vote
1
answer
1k
views
How to create a score for a SWOT analysis (strengths, weaknesses, opportunities, and threats)?
I'm developing a participatory social environmental diagnostic. To do this, I'm using primary (qualitative data from interviews with stakeholders) and secondary data (local socioeconomic data).
From ...
- 113
1
vote
2
answers
4k
views
Which scoring for GridSearchCV is best, when imbalanced multiclass dataset?
I have an unbalanced multiclass dataset (GTSRB) and want to optimize the hyperparameters of an SVM through GridSearchCV.
I know that accuracy is not suitable for scoring in this case.
Which evaluation ...
- 393
3
votes
1
answer
808
views
sklearn.metrics.average_precision_score getting different answers for same data but different formats
I was trying to learn how average precision (AP) is calculated and implemented in scikit-learn. I have read the documentation, but I don't think I fully understand it yet.
Consider the following two ...
- 33
2
votes
1
answer
84
views
what approach to use for find best customer out of data?
I'm working on this project where the objective is to find certain good leads/customers from the existing customer dataset. I tried the RFM method for scoring but there is no data regarding money or ...
- 37
1
vote
0
answers
17
views
Is there a term for measuring error on a second prediction based on the first's?
I have created a dataset which contains six values per row which may be the target value. Two rows for example:
...
- 111
2
votes
1
answer
3k
views
About sklearn.metrics.average_precision_score documentation
There is a example in sklearn.metrics.average_precision_score documentation.
...
- 97
1
vote
0
answers
18
views
Assessing performance of an agent based on commission rate, market share and revenue
I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following:
Number ...
- 85
2
votes
1
answer
88
views
Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)
I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
- 65
4
votes
1
answer
10k
views
How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn
I'm trying to get mean test scores from scikit-learn's GridSearchCV with multiple scorers.
grid.cv_results_ displays lots of ...
- 400
2
votes
1
answer
555
views
How to interpret Sum of Squared Error in a classification task
I am working on ANN. I have 2497 training examples and each of them is a vector of 128, so the input size is 128. Number of neurons in hidden layer is 64 and number of output neurons is 6 (since ...
- 21
4
votes
1
answer
4k
views
Credit scoring using scorecardpy with XGBoost
I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried ...
- 51
3
votes
2
answers
229
views
Is there a definitive and more conclusive way of interpreting the R^2 score from a linear regression model in terms of prediction accuracy?
I'm trying to find a definitive way to conclude the R^2 score from a prediction accuracy point of view rather than variance. How should I do it?
Conceptually, most blogs / articles explain R^2 as:
...
- 31
12
votes
3
answers
2k
views
Why is the F-measure preferred for classification tasks?
Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks?
The F-measure is the ...
- 3,518
4
votes
2
answers
253
views
How can I compare classes from clusterings performed on two different data sets?
I have two data sets defined by real valued vectors, and I have performed clustering on both of them. Now I want to compare the classes to see how they map to each other. If I put the data sets ...
- 143
1
vote
0
answers
93
views
Predict class having only class proportions for every attribute (non labeled data)
I am working with a big data set (millions of observations) where for each observation I am trying to predict a probability (or score it) of being of a class. I haven't any labeled training data and ...
2
votes
1
answer
41
views
How to approach model reporting task
I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates ...
- 254
3
votes
1
answer
69
views
Is the maximum BDeu Bayesian Network always the empty network?
I'm recently reading a paper about Scoring Mechanisms for Bayesian Networks.
For the BDeu score, it appears that the maximum possible score of BDeu for Bayesian Network structure learning is zero.
...
3
votes
1
answer
635
views
Ranking algorithm based on a handful of features
I am trying to determine the apt algorithm for a ranking problem that I am working on. I have social media metrics - engagement, sentiment, audience size etc for several brands and am looking for a ...
- 310