Questions tagged [ranking]
Ranking is ordering data from highest to lowest or *vice versa.* For questions about *constructing* scores to use in ranking, please use the "valuation" tag, too.
118
questions
26
votes
2
answers
28k
views
How fit pairwise ranking models in XGBoost?
As far as I know, to train learning to rank models, you need to have three things in the dataset:
label or relevance
group or query id
feature vector
For example, the Microsoft Learning to Rank ...
- 418
11
votes
1
answer
3k
views
What is query id ("qid") in XGBoost
In XGBoost documentation it's said that for ranking applications we can specify query group ID's qid in the training dataset as in the following snippet:
...
- 153
6
votes
4
answers
10k
views
From pairwise comparisons to ranking - python
I have to solve a ranking ML issue. To start with, I have successfully applied the pointwise ranking approach.
Now, I'm playing around with pairwise ranking algorithms. I've created the pairwise ...
- 300
6
votes
1
answer
16k
views
precision@k and recall@k
Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems.
I ...
- 161
6
votes
2
answers
3k
views
Preparing for a Machine Learning Design Interview
I am not sure if this is a relevant post here but:
I made it to the final round for the Machine Learing Engineer position at Facebook. The final round interview is virtual (thanks to Corona) and will ...
- 237
6
votes
1
answer
3k
views
How to use ndcg metric for binary relevance
I am working on a ranking problem to predict the right single document based on the user query and use the NDCG metric to measure the model.
Given the details :
Queries ( Q ), Result Document ( D ),...
- 61
5
votes
1
answer
264
views
Methods for ensembling ranked lists?
I was wondering if there's a good way to use ensembling when I have two or more algoritims producing ranked lists. That is, suppose I have the following datasets consisting of ordered lists (higher ...
- 151
5
votes
2
answers
2k
views
Converting non-numeric data values into equivalent rank scores
Consider a data-frame similar to the one shown (the actual data-frame is much larger)
...
- 1,783
4
votes
1
answer
2k
views
Can feature importance change a lot between models?
I have a random forest classifier and Multinomial Naive Bayes. For feature importance, I used gini index for random forest and for Multinomial Naive Bayes I used the coefficients of each feature. Then ...
- 155
4
votes
1
answer
96
views
How is "relevance" defined in information retrieval outside the context of systems with user feedback?
I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant".
What can you do if you do not have user ...
- 53
4
votes
5
answers
7k
views
Machine learning algorithm for ranking
I am working on a ranking question, recommending k out of m items to the users. The evaluation metric is average precision at K.
Both R and Python have xgboost can be used for pairwise comparison ...
- 298
4
votes
1
answer
1k
views
Rank players of any given sport
I've recently become interested in possibly of developing some sort of method for ranking athletes of sports such as American football and determining which players are better than others in terms of ...
- 141
4
votes
0
answers
272
views
How Does the Reward Model in ChatGPT Calculate Losses?
Reading the InstructGPT paper(which seems to be what ChatGPT was built off of), I found this equation for the reward function.
However, I'm struggling to understand how this equation is used to ...
- 141
3
votes
2
answers
509
views
Rank feature selection over multiple datasets
Through backward elimination I get a ranking of features over multiple datasets. For example in the dataset 1 I have the following ranking, the feature in the top being the most important:
feat. 1
...
- 863
3
votes
2
answers
216
views
Learning to Rank Application
If there's a website/app that sells products and my job is to determine the order/ranking in which the products should be displayed.
For example : I click on restaurants and a list of restaurants ...
3
votes
1
answer
314
views
Ranking Bias in Learning to Rank
Users tend to click on results ranked highly by search engines much more often than those ranked lower. How do you train a search engine using click data / search logs without this bias? I.e. you don'...
- 133
3
votes
1
answer
137
views
Is it Possible to Use Machine Learning for Ranking Alternatives?
Right now, I’m working on road and street safety analysis. I have a dataset of dangerous points in four regions of a city. Some of the available variables are road lighting status, ITS, latitude, ...
- 31
3
votes
1
answer
83
views
Sparse IR with user feedback
I'm considering a problem framing within an information retrieval context.
I have a sequence of documents that feature different attributes. In the web context, these would be webpages. One ...
3
votes
1
answer
686
views
Ranking algorithm based on a handful of features
I am trying to determine the apt algorithm for a ranking problem that I am working on. I have social media metrics - engagement, sentiment, audience size etc for several brands and am looking for a ...
- 310
3
votes
0
answers
312
views
Why is NDCG high even for wrongly ranked predictions?
The NDCG (Normalized Discounted Cumulative Gain) metric for ranking is defined as DCG/IDCG, where IDCG is the ideal DCG and is said to take values in [0, 1].
However, since the DCG will always be ...
- 131
3
votes
0
answers
751
views
Xgboost rank:ndcg learning per group or for all dataset
I'm trying to implement xgboost with an objective of rank:ndcg
I want the target to be between 0-3.
In my data for most of the groups, there is only 1 event per ...
- 161
2
votes
2
answers
2k
views
XGBoost ranking file format
The xgboost package has two files that must be used for ranking:
train.txt with the data
...
- 3,420
2
votes
1
answer
835
views
Ranking problem and imbalanced dataset
I know about the problems that imbalanced dataset will cause when we are working on classification problems. And I know the solution for that including undersampling and oversampling.
I have to work ...
- 331
2
votes
2
answers
644
views
How to calculate precision at K and NDCG for ranking algorithms
I am ranking a filtered item list as per user's metadata and historical behaviour.
Now how to calculate metrices like precision at K?
One approach could be -
Divide historical data in training and ...
- 71
2
votes
1
answer
268
views
Two definitions of DCG measure
I wanted to check the definition of Discounted Cumulative Gain (DCG) measure in the original paper Jarvelin and it seems it differs from the one given in the later literature Wang. Originally, for $n$ ...
- 343
2
votes
2
answers
363
views
Algorithm Suggestion For a Specific Problem
I'm working on a problem where in I have some data sets about some power generating units. Each of these units have been activated to run in the past and while activation, some units went into some ...
- 137
2
votes
2
answers
417
views
How could PageRank be used to rank paragraphs in relevance to keywords?
I have a data set with keywords describing paragraphs in a car manual and the actual paragraphs. I want to rank those paragraphs by that keyword using PageRank algorithm.
How would I rank these ...
- 21
2
votes
1
answer
295
views
How to determine the "total number of relevant documents" in calculatiion of Recall in Precision and Recall if it's not known? Can it be estimated?
On Wikipedia there is a practical example of calculating Precision and Recall:
When a search engine returns 30 pages, only 20 of which are relevant,
while failing to return 40 additional relevant ...
- 131
2
votes
1
answer
191
views
Learning to Rank with Unlabelled Dataset
I have folder of about 60k PDF documents that I would like to learn to rank based on queries to surface the most relevant results. The goal is to surface and rank relevant documents, very much like a ...
- 51
2
votes
1
answer
487
views
Cosine Similarity but with weighting for vector indexes
I am very new to NLP and although this seems like a basic question I don't know how to search for an answer online.
This is my problem: I have extracted and ranked keywords from 2 text sources:
A ...
- 21
2
votes
1
answer
212
views
Recommender system that matches similar customers with similar highly rated products?
I have a dataset of 1,000 customers that bought 20 distinct phones and rated them 1-5. I have several demographic attributes for these customers (gender, age). My website offers 100 distinct devices, ...
- 181
2
votes
2
answers
2k
views
2
votes
2
answers
176
views
Calculate a ranking function from classification features
I am using 3 features (x1, x2, x3) for binary classification. All my feature values are in 0 to 1 range (unit range).
I obtained how important each feature was in ...
- 133
2
votes
1
answer
222
views
Feature selection with information gain (KL divergence) and mutual information yields different results
I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
- 169
2
votes
1
answer
128
views
Peformance evaluation of ranking algorithms
I have three questions:
How can we assess (or measure) the performance of the ranking algorithms?
Are there any specific measures, or performance metrics, for this?
More specifically, how can we ...
- 121
2
votes
1
answer
63
views
Given rankings and the factors that determined the rankings, what is the best method to reverse engineer the rankings?
I have a list of ranked items and the factors that the group used to create the rankings (like college rankings: 1, 2, 3...), and I'd like to reverse engineer their methodology.
Here's what I tried......
- 21
2
votes
0
answers
80
views
Smallest possible difference between AUC of two ranker [closed]
If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
- 123
2
votes
1
answer
107
views
Drastic drop in Somers' D ? Why?
I came across to find the correlation between the ratings assigned by two coaches to a same group of 40 players.
I have tabulated the results as below:
The Somers' D is 50%.
However, for the case ...
- 21
2
votes
1
answer
259
views
Improve results using user input
I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the ...
- 21
2
votes
1
answer
709
views
What is the difference between nDCG and rank correlation methods?
When do we use one or the other?
My use case:
I want to evaluate a linear space to see how good retrieval results are. I have a set of data X (m x n) and some weights W (m x 1). I want to measure ...
- 21
2
votes
1
answer
105
views
2
votes
0
answers
105
views
Reduce number of vectors in dataset to achieve the "same average dimensions result"? [closed]
Edit for re-opening the question, I'll try to answer questions made by @user2974951:
I have a large user preference statistics for trichotomic data sets. You can visualize each data trio as a 3D ...
- 121
2
votes
0
answers
352
views
Learning ranking
This is a sort of a follow-up to this newbie question:Suppose I want learn ranking (so, I have a bunch of data points, ranked $1, 2, 3, ...$ Now, one way is to use something like logistic regression ...
- 147
1
vote
2
answers
436
views
Use regression instead of classification for hard labeled ranking datasets
Let's imagine I have a dataset of movie reviews with annotated sentiment:
-1 means negative
0 means neutral
+1 means positive
I see a lot of people trying to do ...
- 153
1
vote
1
answer
27
views
What is the best way to pick the optimized configuration from this dataset?
I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, ...
- 113
1
vote
1
answer
251
views
Learning to rank: construct absolute ranking using pair-wise ranking approach
I am learning about the "pairwise approach" for learning to rank. As far as I understood, the training output is a partial ranking function $r$ that:
given given some query $q$ and two document $d_i$ ...
- 163
1
vote
2
answers
487
views
How to interpret rank output from BM25?
I'm using the FTS5 functionality in Sqlite3 to compute a search rank, which uses BM25. Maybe I'm just not seeing it, but I can't find any description of how to actually interpret BM25's output ranking....
- 121
1
vote
1
answer
103
views
Use dummy variables to create a rank variable. R
I have a series of multiple response (dummy) variables describing causes for a canceled visits. A visit can have multiple reasons for the cancelation. My goal is to create a single mutually exclusive ...
- 37
1
vote
1
answer
31
views
Changing behaviour of an ML model
I am trying to create a ranking system for recommending books to an user. Let's suppose we have some subjects of books like 'A', 'B', 'C', 'D' and from the past behaviour, it is observed that the user ...
- 189
1
vote
1
answer
371
views
Is there a good alternative to XGBoost for learn to rank?
My problem with XGBoost is that when I load the train dataset into the XGBoost DMatrix, there is a memory spike that is unavoidable, and I can't get my dataset loaded into RAM without crashing first.
...
- 55