# Reducing the effect of down voters with rating system

I have a site in which users rate things in a 1-5 star system. Once an item reaches the top of the charts, some users tend to start rating it 1 star even though it got a majority of 4-5 stars to get where it's at. It's not rampant, I would say 10-20% of the new votes are 1's. Clearly they are trying to manipulate the rating system, and I want to prevent that.

The current way I am doing that is by having a "reasonable window" of what I consider to be a legitimate vote.

For items with less than 10 votes; I currently do nothing and take the mean as it's rating.

Once an item starts getting more than 10 votes I tie them to a window of their mean. This window is defined as

Window = 4.5 - Log(TotalVotes, 10);


So a reasonable vote range is then (Mean - Window) thru (Mean + Window)

Once the reasonable vote range is found, the "Rating" is just the mean of all the reasonable votes (those who fall in the reasonable range).

This means an item with a real mean of 4.2 with 100 votes would have a window of 4.5-Log(100,10) = 2.5, so if that item gets a 1 star vote, it will be ignored in the rating. However, the 1 star will still effect the underlying mean.

This has worked good in general but the issue is when an item's Mean - Window is just at the brink of 1.0, as soon as it dips below 1.0 every 1 star vote now is included in the rating and the rating drops significantly even the difference before and after may have just been one more 1 star rating.

I need a better system/way of accomplishing to filter out these 1 star ratings, and not just them but handle the situation in where someone may get their friends to upvote an item 10 votes and all 5 stars, where its true rating may be more 3 star.

Looking for any recommendations of how to handle user-driven rating systems and normalizing outlier votes.

• You could give later votes (good or bad) a lower weight. But then someone that gets friend to vote early will get a higher weight. – paparazzo Nov 28 '15 at 14:12
• There's a good analogue to this in time series analysis- smoothing is accomplished via a moving window. Windows were once step functions (just like your "window"), but perform much better when they are gaussians. You should turn your window into a gaussian and weight the votes by the value of the gaussian. I'll try to turn this into an answer, but am really busy now. Check out exponentially weighted moving average (EWMA) but don't move it i.e. EWA. Its nice since everything becomes smooth and you get away from the quantum jumps from contributing scores to non-contributing scores. – AN6U5 Nov 28 '15 at 18:24
• What if there is a reason for that? Say in Google Play Store, a new version of an app is uploaded. And oops, it is broken. Of course a lot of people will give a 1 star rating then, without being not reasonable. I would look for other indicators of manipulation. – Has QUIT--Anony-Mousse Nov 29 '15 at 11:13

You should look into other estimators of location.

What you want is a robust estimator, with a high break-down point.

The extreme approach would be the median.

But you may get more numerically interesting results with a trimmed mean.

You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of the remaining entries. An app with 98% 5 stars will still get a 5.0

But to prevent manipulation, I would look into other signals. Such as clustered votes from a single region, for example.

• Thanks for the recommendation, glad to know the technical terms of what I am after. The issue with a trimmed mean is that outfits who never appear on the first page almost get no 1-votes, whereas those on the front page get 10% or more. If I trim the top and bottom 2% the problem is still there for front-pagers and it hurts second-pagers greatly. To prevent manipulation I do delete votes that seem far from the mean when grouped by IP and geographic region. However this is a clean up task I run every few months, what is in question here is how to handle it on the fly – ParoX Nov 29 '15 at 18:44
• If you do this on-the-fly as you discussed above, you become much slower. What if there is a reason many people start to give low marks (e.g. because it has become known the product has a problem?) – Has QUIT--Anony-Mousse Nov 29 '15 at 19:52

I like @Anony-Mousse's answer. Using robust estimators is good.

I want to add a different direction to cope with the problem. It seems that there are some "malicious" users casting these down votes so you might want to identify them.

Create a dataset of the users and use "casted unjustified down vote on leading item" as the label. You can use "casted down vote on leading item" as the default value and then manually modify them and make the rule more delicate like "casted more than twice down vote on leading item after the item reached the top charts" I guess that features like number of low votes, number of low votes to leading items, etc will be useful.

Now you are in a supervised learning framework. Once you identify malicious users, ignore their votes and avoid the manipulations.

To robustify your estimator, you might model your ratings as a Gaussian mixture model (GMM) that is a mixture of two Gaussian rvs: 1) true ratings, 2) junk rating that are equal to one. Scikit-learn already has a canned GMM classifier: http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_classifier.html#example-mixture-plot-gmm-classifier-py

Digging in a little more, a simple approach would be to let scikit-learn partition your ratings into two gaussians. If one of the partitions ends up with a mean near one, then we can throw out those ratings. Or, more elegantly, we can take the mean of the other, non-near-one Gaussian, as the true rating mean.

Here is a bit of code for a ipython notebook that does this:

from sklearn.mixture import GMM
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import collections

def make_ratings(mean,std,rating_cnt):
rating_sample = np.random.randn(rating_cnt)*std + mean
return np.clip(rating_sample,1,5).astype(int)

def make_collection(true_mean,true_std,true_cnt,junk_count):
true_ratings = make_ratings(true_mean,true_std,true_cnt)
junk_ratings = make_ratings(1,0,junk_count)
return np.hstack([true_ratings,junk_ratings])[:,np.newaxis]

def robust_mean(X, th = 2.5, agg_th=2.5, default_agg=np.mean):
classifier = GMM(n_components=2)
classifier.fit(X)
if np.min(classifier.means_) > th or default_agg(X)<agg_th:
return default_agg(X)
else:
return np.max(classifier.means_)

r_mean = 4.2
X = make_collection(r_mean,2,40,10)
plt.hist(X,5)
classifier = GMM(n_components=2)
classifier.fit(X)
plt.show()
print "vars =",classifier.covars_.flatten()
print "means = ",classifier.means_.flatten()
print "mean = ",np.mean(X)
print "median = ",np.median(X)
print "robust mean = ", robust_mean(X)
print "true mean = ", r_mean
print "prob(rating=1|class) = ",classifier.predict_proba(1).flatten()
print "prob(rating=true_mean|class) = ",classifier.predict_proba(r_mean).flatten()
print "prediction: ", classifier.predict(X)


The output for one run looks like:

vars = [ 0.22386589  0.56931527]
means =  [ 1.32310978  4.00603523]
mean =  2.9
median =  3.0
robust mean =  4.00603523034
true mean =  4.2
prob(rating=1|class) =  [  9.99596493e-01   4.03507425e-04]
prob(rating=true_mean|class) =  [  1.08366762e-08   9.99999989e-01]
prediction:  [1 0 1 0 1 1 1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 1 0 1 0 1 0 1 1 0
1 1 1 0 0 0 0 0 0 0 0 0 0]


We can simulate how will this works with a few monte carlo trials:

true_means = np.arange(1.5,4.5,.2)
true_ratings = 40
junk_ratings = 10
true_std = 1
m_out = []
m_in = []
m_reg = []
runs = 40
for m in true_means:
Xs = [make_collection(m,true_std,true_ratings,junk_ratings) for x in range(runs)]
m_in.append([[m]*runs])
m_out.append([[robust_mean(X, th = 2.5, agg_th=2,default_agg=np.mean) for X in Xs]])
m_reg.append([[np.mean(X) for X in Xs]])

m_in = np.array(m_in).T[:,0,:]
m_out = np.array(m_out).T[:,0,:]
m_reg = np.array(m_reg).T[:,0,:]

plt.plot(m_in,m_out,'b.',alpha=.25)
plt.plot(m_in,m_reg,'r.',alpha=.25)
plt.plot(np.arange(0,5,.1),np.arange(0,5,.1),'k.')
plt.xlim([0,5])
plt.ylim([0,5])
plt.xlabel('true mean')
plt.ylabel('predicted mean')
plt.title("true_ratings=" + str(true_ratings)
+ "; junk_ratings=" + str(junk_ratings)
+ "; std="+str(true_std))


The output is pasted below. The red is the mean rating and the blue is the proposed rating. You can tweak the parameters to get slightly different behaviors.