Questions tagged [scoring]
The scoring tag has no usage guidance.
37
questions
0
votes
1answer
14 views
How to compute AUC in gridsearchSV (multiclass problem)
I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the ...
0
votes
0answers
12 views
scaling credit risk scorecard
I need to build a credit risk scorecard using logistic and linear regression. The variables using to predict are all dummies, where each dummy is a bin of some variable.
Let's say the variable age, I ...
2
votes
0answers
23 views
Random forest mode scoring
We are using random forest algorithm but having some trouble understanding the scoring method it uses.
take for example the following CM of the test set:
...
0
votes
1answer
20 views
0
votes
0answers
21 views
Scoring products across all stores
I am trying to score each product based off a couple of features. I was thinking I can use the sales rate(7 days prior) as the target column and predict the sales rate using some ensemble technique ...
2
votes
0answers
33 views
Best practices for scoring hundreds of models on the same massive dataset?
I have 500+ models predicting various things and a massive database of over 400m+ individuals and about 5,000 possible independent variables.
Currently, my scoring process takes about 5 days, and ...
0
votes
1answer
85 views
How to create a score for a SWOT analysis (strengths, weaknesses, opportunities, and threats)?
I'm developing a participatory social environmental diagnostic. To do this, I'm using primary (qualitative data from interviews with stakeholders) and secondary data (local socioeconomic data).
From ...
1
vote
2answers
199 views
Which scoring for GridSearchCV is best, when imbalanced multiclass dataset?
I have an unbalanced multiclass dataset (GTSRB) and want to optimize the hyperparameters of an SVM through GridSearchCV.
I know that accuracy is not suitable for scoring in this case.
Which evaluation ...
3
votes
1answer
103 views
sklearn.metrics.average_precision_score getting different answers for same data but different formats
I was trying to learn how average precision (AP) is calculated and implemented in scikit-learn. I have read the documentation, but I don't think I fully understand it yet.
Consider the following two ...
2
votes
1answer
41 views
what approach to use for find best customer out of data?
I'm working on this project where the objective is to find certain good leads/customers from the existing customer dataset. I tried the RFM method for scoring but there is no data regarding money or ...
0
votes
0answers
137 views
For a multiclass classification problem, how do we find the cohen kappa score?
So I have a multiclass classification problem and I have found the Matthews Correlation Coefficient of that (https://scikit-learn.org/stable/modules/model_evaluation.html#matthews-correlation-...
0
votes
0answers
39 views
Open Set Classification : How to count True Positive, False Positive,
TL;DR : How to compute TP,FP,TN,FN in Open Set Classification setting ?
Even if the problem is simple; the answer may be tricky, so is my question.
Given two sets of clusters : one from experiments, ...
0
votes
0answers
25 views
Is the precision_recall_curve reasonable?
I always get this 'pattern' PR curve when I use precision_recall_curve function to plot PR curve.
The starting point is always from (0,1).
Because when thresholds set up the highest score lend to ...
1
vote
0answers
17 views
Is there a term for measuring error on a second prediction based on the first's?
I have created a dataset which contains six values per row which may be the target value. Two rows for example:
...
2
votes
1answer
545 views
About sklearn.metrics.average_precision_score documentation
There is a example in sklearn.metrics.average_precision_score documentation.
...
1
vote
0answers
16 views
Assessing performance of an agent based on commission rate, market share and revenue
I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following:
Number ...
2
votes
1answer
59 views
Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)
I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
1
vote
1answer
3k views
How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn
I'm trying to get mean test scores from scikit-learn's GridSearchCV with multiple scorers.
grid.cv_results_ displays lots of ...
4
votes
1answer
1k views
Credit scoring using scorecardpy with XGBoost
I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried ...
3
votes
2answers
93 views
Is there a definitive and more conclusive way of interpreting the R^2 score from a linear regression model in terms of prediction accuracy?
I'm trying to find a definitive way to conclude the R^2 score from a prediction accuracy point of view rather than variance. How should I do it?
Conceptually, most blogs / articles explain R^2 as:
...
8
votes
2answers
647 views
Why is the F-measure preferred for classification tasks?
Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks?
The F-measure is the ...
4
votes
2answers
48 views
How can I compare classes from clusterings performed on two different data sets?
I have two data sets defined by real valued vectors, and I have performed clustering on both of them. Now I want to compare the classes to see how they map to each other. If I put the data sets ...
1
vote
0answers
67 views
Predict class having only class proportions for every attribute (non labeled data)
I am working with a big data set (millions of observations) where for each observation I am trying to predict a probability (or score it) of being of a class. I haven't any labeled training data and ...
2
votes
1answer
29 views
How to approach model reporting task
I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates ...
2
votes
1answer
42 views
Is the maximum BDeu Bayesian Network always the empty network?
I'm recently reading a paper about Scoring Mechanisms for Bayesian Networks.
For the BDeu score, it appears that the maximum possible score of BDeu for Bayesian Network structure learning is zero.
...
2
votes
0answers
390 views
Ranking algorithm based on a handful of features
I am trying to determine the apt algorithm for a ranking problem that I am working on. I have social media metrics - engagement, sentiment, audience size etc for several brands and am looking for a ...
1
vote
0answers
55 views
What is an appropriate way to compare classifiers with different sets of classes?
I have three classifiers for language identification:
A: en, de, ru, fr, ij, kl
B: en, de, ru, fr, xy
C: en, de, ru, fr, no, pq, rs
and I have a balanced dataset ...
2
votes
0answers
26 views
Intuitive interpretation of ratios between training set scores and validation set scores
I'm training models with the usual setup where you hold back a portion (in my case, 20%) of the data just to see how your trained model generalizes to unseen data, to see if it's overfitting.
When ...
2
votes
3answers
85 views
How to learn to score new documents based on a existing set of scored documents?
I have a 50 000 documents of 1000 words or more ranked between 0 and 2000. They all deal with a similar topic. I'd like to create an algorithm that can learn to score new documents.
What approach do ...
3
votes
1answer
880 views
Selecting the right algorithm for match probability prediction
Looking for assistance kick-starting a new machine learning scenario. In this case I need to pair one entity (ex. person) with a group of entities (ex. other people) given a history of matching ...
5
votes
2answers
1k views
Converting non-numeric data values into equivalent rank scores
Consider a data-frame similar to the one shown (the actual data-frame is much larger)
...
4
votes
1answer
946 views
Prediction model for marketing to prospective customers (using pandas)
I'm currently working on a part-time project which involves predicting the likelihood of customers going to buy a product using data analytics. The company I'm interning with has given me a customer ...
7
votes
2answers
1k views
Why are precision and recall used in the F1 score, rather than precision and NPV?
In binary classification problems it seems the F1 score is often used as a performance measure. As far as I've understood the idea is to find the best tradeoff between precision and recall. The ...
2
votes
2answers
139 views
Creating validation data for model comparison
I am working on building a scoring algorithm for student data,
say the attributes are :
name, location, age, class, school_name, skill1, skill2, skill3
based ...
5
votes
1answer
441 views
Comparing accuracy of models in ordinal regression / classification
I am looking into creating a model to predict whether an item is "Very Good", "Good", "Bad" or "Very Bad".
After I fit the training data to the models, comparing the accuracy of the models during ...
3
votes
2answers
1k views
R in production
Many of us are very familiar with using R in reproducible, but very much targeted, ad-hoc analysis. Given that R is currently the best collection of cutting-edge scientific methods from world-class ...
4
votes
1answer
604 views
Rank players of any given sport
I've recently become interested in possibly of developing some sort of method for ranking athletes of sports such as American football and determining which players are better than others in terms of ...
