Say you have an organization that requires employees to participate in a Q&A site similar to StackOverflow - questions and answers are voted upon, selected answers get extra points, certain behaviors boost your score etc. What we need to do is assign a rating from 1-100 to these users with even distribution.
The behaviors that add points:
- Ask a question [fixed]
- Answer a question [fixed]
- Receive an upvote on a question [determined by relative ranking]
- Receive an upvote on an answer [determined by relative ranking]
- Have your answer selected [determined by relative ranking]
- Responding to a comment, etc [fixed]
Likewise, there are behaviors that subtract points.
If a user with a high ranking upvotes a question asked by a lower-ranking user, more points should be awarded than the inverse situation. Likewise if a lower-ranking user downvotes a higher-ranking user's question, the impact should be minimal compared to the inverse. There should be a limit to this impact though so that a high-ranking user doesn't unintentionally destroy any momentum of a low-ranking user by issuing a powerful downvote.
We have a few challenges here:
How do we determine how many points to assign to each type of behavior, with actor/recipient relative rank taken into account?
I'm thinking we just assign a flat number to each behavior, that number decided relative to the importance of the other behaviors, and then have a variable score that can alter the score if there is a wide variance between the users. The mechanics of this - does the score double at most? - are unclear.
How to we assign this rank? This one is a little easier - I'm thinking we just order the users according to score and then split the dataset into 100 sections, assigning each "chunk" a number 1-100.
Should we be worried about these numbers getting "very big"? The scenario described above has been trivialized; actions taken by these users may happen hundreds of times per day so the scores can become very high, very quickly. Is there a way we can keep this under control while avoiding a large number of duplicate scores?
How do we define the "fixed" scores as the total scores become very big? Over time we may have users with hundreds of thousands of points - but the fixed-score behaviors should still reward them. They should reward lower-ranking users more than higher-ranking users.
I don't know if there are some standard practices, algorithms, or terminologies that I should be aware of when facing a problem like this - any input would be appreciated.