Methods / Algorithms for rank scales based on cumulative scoring

Question

Say you have an organization that requires employees to participate in a Q&A site similar to StackOverflow - questions and answers are voted upon, selected answers get extra points, certain behaviors boost your score etc. What we need to do is assign a rating from 1-100 to these users with even distribution.

The behaviors that add points:

• Ask a question [fixed]
• Answer a question [fixed]
• Receive an upvote on a question [determined by relative ranking]
• Receive an upvote on an answer [determined by relative ranking]
• Have your answer selected [determined by relative ranking]
• Responding to a comment, etc [fixed]

Likewise, there are behaviors that subtract points.

If a user with a high ranking upvotes a question asked by a lower-ranking user, more points should be awarded than the inverse situation. Likewise if a lower-ranking user downvotes a higher-ranking user's question, the impact should be minimal compared to the inverse. There should be a limit to this impact though so that a high-ranking user doesn't unintentionally destroy any momentum of a low-ranking user by issuing a powerful downvote.

We have a few challenges here:

1. How do we determine how many points to assign to each type of behavior, with actor/recipient relative rank taken into account?
   I'm thinking we just assign a flat number to each behavior, that number decided relative to the importance of the other behaviors, and then have a variable score that can alter the score if there is a wide variance between the users. The mechanics of this - does the score double at most? - are unclear.

2. How to we assign this rank? This one is a little easier - I'm thinking we just order the users according to score and then split the dataset into 100 sections, assigning each "chunk" a number 1-100.

3. Should we be worried about these numbers getting "very big"? The scenario described above has been trivialized; actions taken by these users may happen hundreds of times per day so the scores can become very high, very quickly. Is there a way we can keep this under control while avoiding a large number of duplicate scores?

4. How do we define the "fixed" scores as the total scores become very big? Over time we may have users with hundreds of thousands of points - but the fixed-score behaviors should still reward them. They should reward lower-ranking users more than higher-ranking users.

I don't know if there are some standard practices, algorithms, or terminologies that I should be aware of when facing a problem like this - any input would be appreciated.

Answer 1

To solve challenges #3 and #4, let's limit the overall available rank volume. For example, sum of this rank for all the users will be 1 (100%).

From challenge #2 I understood, that you accept 2 different ranks: (1) place from 1 to 100, and (2) simple sum of all earned points (fixed and relative). Did I got it right? I so, there is no need to worry about unlimited growth, or fixed scores inflation. Let's just use percentages, not 1-100 ranks.

These percentage ranks could be calculated based on interaction behaviors (vote/selecting answer/etc), using PageRank-like algorithm. Such algorithm will consider all previous reactions (and ranks of acted users), obtained by an exact user. Unfortunately, you cannot use PageRank algorithm "as is", because it supports only "positive" links, but you can look for it's extensions. For example, look at this paper with PageRank extension for both positive and negative links (as users can down vote). You can iteratively estimate percentage rank (TrustRank, TR) using this algorithm.

The second task is to calculate reward/penalty rate in points for each single action. Let's determine (predefine) maximal reward/penalty rate (X) for each type of action. And will use coefficient to discount it, based on TrustRanks of acting users (e.g., author and voter). Slightly modified Sigmoid will map this ratio from [-Inf,+Inf] range to [0,1].

Here for peer users you will have ~0.5 of predefined maximal rate. If "voter" has TR twice more than "author", "author" will recieve ~0.75 of predefined value, and so on. You can tune steepness with additional parameter, or try to find any other mapping transformation function.

Anyway, now simply multiply maximal penalty/reward by this coefficient, and you'll get the number of point, you need to deduct or add. The only issue, I see, is a user with zero TR - such user as a voter will "give" nothing, and as an object of voting, will recieve the maximal amount of points regardless voter's rank. To avoid this, you can predefine minimal TR (like 1e-10), and don't let user's TR to fall beyond this value.