Reducing the effect of down voters with rating system

Question

I have a site in which users rate things in a 1-5 star system. Once an item reaches the top of the charts, some users tend to start rating it 1 star even though it got a majority of 4-5 stars to get where it's at. It's not rampant, I would say 10-20% of the new votes are 1's. Clearly they are trying to manipulate the rating system, and I want to prevent that.

The current way I am doing that is by having a "reasonable window" of what I consider to be a legitimate vote.

For items with less than 10 votes; I currently do nothing and take the mean as it's rating.

Once an item starts getting more than 10 votes I tie them to a window of their mean. This window is defined as

Window = 4.5 - Log(TotalVotes, 10);

So a reasonable vote range is then (Mean - Window) thru (Mean + Window)

Once the reasonable vote range is found, the "Rating" is just the mean of all the reasonable votes (those who fall in the reasonable range).

This means an item with a real mean of 4.2 with 100 votes would have a window of 4.5-Log(100,10) = 2.5, so if that item gets a 1 star vote, it will be ignored in the rating. However, the 1 star will still effect the underlying mean.

This has worked good in general but the issue is when an item's Mean - Window is just at the brink of 1.0, as soon as it dips below 1.0 every 1 star vote now is included in the rating and the rating drops significantly even the difference before and after may have just been one more 1 star rating.

I need a better system/way of accomplishing to filter out these 1 star ratings, and not just them but handle the situation in where someone may get their friends to upvote an item 10 votes and all 5 stars, where its true rating may be more 3 star.

Looking for any recommendations of how to handle user-driven rating systems and normalizing outlier votes.

Answer 1

You should look into other estimators of location.

What you want is a robust estimator, with a high break-down point.

The extreme approach would be the median.

But you may get more numerically interesting results with a trimmed mean.

You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of the remaining entries. An app with 98% 5 stars will still get a 5.0

But to prevent manipulation, I would look into other signals. Such as clustered votes from a single region, for example.

Answer 2

I like @Anony-Mousse's answer. Using robust estimators is good.

I want to add a different direction to cope with the problem. It seems that there are some "malicious" users casting these down votes so you might want to identify them.

Create a dataset of the users and use "casted unjustified down vote on leading item" as the label. You can use "casted down vote on leading item" as the default value and then manually modify them and make the rule more delicate like "casted more than twice down vote on leading item after the item reached the top charts" I guess that features like number of low votes, number of low votes to leading items, etc will be useful.

Now you are in a supervised learning framework. Once you identify malicious users, ignore their votes and avoid the manipulations.

Answer 3

To robustify your estimator, you might model your ratings as a Gaussian mixture model (GMM) that is a mixture of two Gaussian rvs: 1) true ratings, 2) junk rating that are equal to one. Scikit-learn already has a canned GMM classifier: http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_classifier.html#example-mixture-plot-gmm-classifier-py

Digging in a little more, a simple approach would be to let scikit-learn partition your ratings into two gaussians. If one of the partitions ends up with a mean near one, then we can throw out those ratings. Or, more elegantly, we can take the mean of the other, non-near-one Gaussian, as the true rating mean.

Here is a bit of code for a ipython notebook that does this:

from sklearn.mixture import GMM
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import collections

def make_ratings(mean,std,rating_cnt):
    rating_sample = np.random.randn(rating_cnt)*std + mean
    return np.clip(rating_sample,1,5).astype(int)

def make_collection(true_mean,true_std,true_cnt,junk_count):
    true_ratings = make_ratings(true_mean,true_std,true_cnt)
    junk_ratings = make_ratings(1,0,junk_count)
    return np.hstack([true_ratings,junk_ratings])[:,np.newaxis]

def robust_mean(X, th = 2.5, agg_th=2.5, default_agg=np.mean):
    classifier = GMM(n_components=2)
    classifier.fit(X)
    if np.min(classifier.means_) > th or default_agg(X)<agg_th:
        return default_agg(X)
    else:
        return np.max(classifier.means_)

r_mean = 4.2
X = make_collection(r_mean,2,40,10)
plt.hist(X,5)
classifier = GMM(n_components=2)
classifier.fit(X)
plt.show()
print "vars =",classifier.covars_.flatten()
print "means = ",classifier.means_.flatten()
print "mean = ",np.mean(X)
print "median = ",np.median(X)
print "robust mean = ", robust_mean(X)
print "true mean = ", r_mean
print "prob(rating=1|class) = ",classifier.predict_proba(1).flatten()
print "prob(rating=true_mean|class) = ",classifier.predict_proba(r_mean).flatten()
print "prediction: ", classifier.predict(X)

The output for one run looks like:

vars = [ 0.22386589  0.56931527]
means =  [ 1.32310978  4.00603523]
mean =  2.9
median =  3.0
robust mean =  4.00603523034
true mean =  4.2
prob(rating=1|class) =  [  9.99596493e-01   4.03507425e-04]
prob(rating=true_mean|class) =  [  1.08366762e-08   9.99999989e-01]
prediction:  [1 0 1 0 1 1 1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 1 0 1 0 1 0 1 1 0
 1 1 1 0 0 0 0 0 0 0 0 0 0]

We can simulate how will this works with a few monte carlo trials:

true_means = np.arange(1.5,4.5,.2)
true_ratings = 40
junk_ratings = 10
true_std = 1
m_out = []
m_in = []
m_reg = []
runs = 40
for m in true_means:
    Xs = [make_collection(m,true_std,true_ratings,junk_ratings) for x in range(runs)]
    m_in.append([[m]*runs])
    m_out.append([[robust_mean(X, th = 2.5, agg_th=2,default_agg=np.mean) for X in Xs]])
    m_reg.append([[np.mean(X) for X in Xs]])

m_in = np.array(m_in).T[:,0,:]
m_out = np.array(m_out).T[:,0,:]
m_reg = np.array(m_reg).T[:,0,:]

plt.plot(m_in,m_out,'b.',alpha=.25)
plt.plot(m_in,m_reg,'r.',alpha=.25)
plt.plot(np.arange(0,5,.1),np.arange(0,5,.1),'k.')
plt.xlim([0,5])
plt.ylim([0,5])
plt.xlabel('true mean')
plt.ylabel('predicted mean')
plt.title("true_ratings=" + str(true_ratings)
          + "; junk_ratings=" + str(junk_ratings)
         + "; std="+str(true_std))

The output is pasted below. The red is the mean rating and the blue is the proposed rating. You can tweak the parameters to get slightly different behaviors.

Answer 4

Record all votes

Ratio number of 1 votes when on the front page compared to not

Only apply a fraction of number 1 votes while on the first page
Basically remove the page 1 bias based on the page 1 bias as a whole

1 vote applied = 1 votes item when on first page * (1 votes second page total / 1 votes first page total)

Answer 5

While it is technically probably easiest to implement one of the above solutions, I think you should also consider disincentivising the voters to downvote. For example, if the downvotes come from a minority of users who are clearly abusing the system, their repeated downvotes should count (negatively) toward their reputation - like this site.