7
$\begingroup$

I have a site in which users rate things in a 1-5 star system. Once an item reaches the top of the charts, some users tend to start rating it 1 star even though it got a majority of 4-5 stars to get where it's at. It's not rampant, I would say 10-20% of the new votes are 1's. Clearly they are trying to manipulate the rating system, and I want to prevent that.

The current way I am doing that is by having a "reasonable window" of what I consider to be a legitimate vote.

For items with less than 10 votes; I currently do nothing and take the mean as it's rating.

Once an item starts getting more than 10 votes I tie them to a window of their mean. This window is defined as

Window = 4.5 - Log(TotalVotes, 10);

So a reasonable vote range is then (Mean - Window) thru (Mean + Window)

Once the reasonable vote range is found, the "Rating" is just the mean of all the reasonable votes (those who fall in the reasonable range).

This means an item with a real mean of 4.2 with 100 votes would have a window of 4.5-Log(100,10) = 2.5, so if that item gets a 1 star vote, it will be ignored in the rating. However, the 1 star will still effect the underlying mean.

This has worked good in general but the issue is when an item's Mean - Window is just at the brink of 1.0, as soon as it dips below 1.0 every 1 star vote now is included in the rating and the rating drops significantly even the difference before and after may have just been one more 1 star rating.

I need a better system/way of accomplishing to filter out these 1 star ratings, and not just them but handle the situation in where someone may get their friends to upvote an item 10 votes and all 5 stars, where its true rating may be more 3 star.

Looking for any recommendations of how to handle user-driven rating systems and normalizing outlier votes.

$\endgroup$
  • $\begingroup$ You could give later votes (good or bad) a lower weight. But then someone that gets friend to vote early will get a higher weight. $\endgroup$ – paparazzo Nov 28 '15 at 14:12
  • $\begingroup$ There's a good analogue to this in time series analysis- smoothing is accomplished via a moving window. Windows were once step functions (just like your "window"), but perform much better when they are gaussians. You should turn your window into a gaussian and weight the votes by the value of the gaussian. I'll try to turn this into an answer, but am really busy now. Check out exponentially weighted moving average (EWMA) but don't move it i.e. EWA. Its nice since everything becomes smooth and you get away from the quantum jumps from contributing scores to non-contributing scores. $\endgroup$ – AN6U5 Nov 28 '15 at 18:24
  • 2
    $\begingroup$ What if there is a reason for that? Say in Google Play Store, a new version of an app is uploaded. And oops, it is broken. Of course a lot of people will give a 1 star rating then, without being not reasonable. I would look for other indicators of manipulation. $\endgroup$ – Anony-Mousse Nov 29 '15 at 11:13
5
$\begingroup$

You should look into other estimators of location.

What you want is a robust estimator, with a high break-down point.

The extreme approach would be the median.

But you may get more numerically interesting results with a trimmed mean.

You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of the remaining entries. An app with 98% 5 stars will still get a 5.0

But to prevent manipulation, I would look into other signals. Such as clustered votes from a single region, for example.

$\endgroup$
  • $\begingroup$ Thanks for the recommendation, glad to know the technical terms of what I am after. The issue with a trimmed mean is that outfits who never appear on the first page almost get no 1-votes, whereas those on the front page get 10% or more. If I trim the top and bottom 2% the problem is still there for front-pagers and it hurts second-pagers greatly. To prevent manipulation I do delete votes that seem far from the mean when grouped by IP and geographic region. However this is a clean up task I run every few months, what is in question here is how to handle it on the fly $\endgroup$ – ParoX Nov 29 '15 at 18:44
  • 2
    $\begingroup$ If you do this on-the-fly as you discussed above, you become much slower. What if there is a reason many people start to give low marks (e.g. because it has become known the product has a problem?) $\endgroup$ – Anony-Mousse Nov 29 '15 at 19:52
1
$\begingroup$

I like @Anony-Mousse's answer. Using robust estimators is good.

I want to add a different direction to cope with the problem. It seems that there are some "malicious" users casting these down votes so you might want to identify them.

Create a dataset of the users and use "casted unjustified down vote on leading item" as the label. You can use "casted down vote on leading item" as the default value and then manually modify them and make the rule more delicate like "casted more than twice down vote on leading item after the item reached the top charts" I guess that features like number of low votes, number of low votes to leading items, etc will be useful.

Now you are in a supervised learning framework. Once you identify malicious users, ignore their votes and avoid the manipulations.

$\endgroup$
1
$\begingroup$

To robustify your estimator, you might model your ratings as a Gaussian mixture model (GMM) that is a mixture of two Gaussian rvs: 1) true ratings, 2) junk rating that are equal to one. Scikit-learn already has a canned GMM classifier: http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_classifier.html#example-mixture-plot-gmm-classifier-py

Digging in a little more, a simple approach would be to let scikit-learn partition your ratings into two gaussians. If one of the partitions ends up with a mean near one, then we can throw out those ratings. Or, more elegantly, we can take the mean of the other, non-near-one Gaussian, as the true rating mean.

Here is a bit of code for a ipython notebook that does this:

from sklearn.mixture import GMM
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import collections

def make_ratings(mean,std,rating_cnt):
    rating_sample = np.random.randn(rating_cnt)*std + mean
    return np.clip(rating_sample,1,5).astype(int)

def make_collection(true_mean,true_std,true_cnt,junk_count):
    true_ratings = make_ratings(true_mean,true_std,true_cnt)
    junk_ratings = make_ratings(1,0,junk_count)
    return np.hstack([true_ratings,junk_ratings])[:,np.newaxis]

def robust_mean(X, th = 2.5, agg_th=2.5, default_agg=np.mean):
    classifier = GMM(n_components=2)
    classifier.fit(X)
    if np.min(classifier.means_) > th or default_agg(X)<agg_th:
        return default_agg(X)
    else:
        return np.max(classifier.means_)

r_mean = 4.2
X = make_collection(r_mean,2,40,10)
plt.hist(X,5)
classifier = GMM(n_components=2)
classifier.fit(X)
plt.show()
print "vars =",classifier.covars_.flatten()
print "means = ",classifier.means_.flatten()
print "mean = ",np.mean(X)
print "median = ",np.median(X)
print "robust mean = ", robust_mean(X)
print "true mean = ", r_mean
print "prob(rating=1|class) = ",classifier.predict_proba(1).flatten()
print "prob(rating=true_mean|class) = ",classifier.predict_proba(r_mean).flatten()
print "prediction: ", classifier.predict(X)

The output for one run looks like:

vars = [ 0.22386589  0.56931527]
means =  [ 1.32310978  4.00603523]
mean =  2.9
median =  3.0
robust mean =  4.00603523034
true mean =  4.2
prob(rating=1|class) =  [  9.99596493e-01   4.03507425e-04]
prob(rating=true_mean|class) =  [  1.08366762e-08   9.99999989e-01]
prediction:  [1 0 1 0 1 1 1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 1 0 1 0 1 0 1 1 0
 1 1 1 0 0 0 0 0 0 0 0 0 0]

We can simulate how will this works with a few monte carlo trials:

true_means = np.arange(1.5,4.5,.2)
true_ratings = 40
junk_ratings = 10
true_std = 1
m_out = []
m_in = []
m_reg = []
runs = 40
for m in true_means:
    Xs = [make_collection(m,true_std,true_ratings,junk_ratings) for x in range(runs)]
    m_in.append([[m]*runs])
    m_out.append([[robust_mean(X, th = 2.5, agg_th=2,default_agg=np.mean) for X in Xs]])
    m_reg.append([[np.mean(X) for X in Xs]])

m_in = np.array(m_in).T[:,0,:]
m_out = np.array(m_out).T[:,0,:]
m_reg = np.array(m_reg).T[:,0,:]

plt.plot(m_in,m_out,'b.',alpha=.25)
plt.plot(m_in,m_reg,'r.',alpha=.25)
plt.plot(np.arange(0,5,.1),np.arange(0,5,.1),'k.')
plt.xlim([0,5])
plt.ylim([0,5])
plt.xlabel('true mean')
plt.ylabel('predicted mean')
plt.title("true_ratings=" + str(true_ratings)
          + "; junk_ratings=" + str(junk_ratings)
         + "; std="+str(true_std))

The output is pasted below. The red is the mean rating and the blue is the proposed rating. You can tweak the parameters to get slightly different behaviors. enter image description here

$\endgroup$
0
$\begingroup$

Record all votes

Ratio number of 1 votes when on the front page compared to not

Only apply a fraction of number 1 votes while on the first page
Basically remove the page 1 bias based on the page 1 bias as a whole

1 vote applied = 1 votes item when on first page * (1 votes second page total / 1 votes first page total)

$\endgroup$
0
$\begingroup$

While it is technically probably easiest to implement one of the above solutions, I think you should also consider disincentivising the voters to downvote. For example, if the downvotes come from a minority of users who are clearly abusing the system, their repeated downvotes should count (negatively) toward their reputation - like this site.

$\endgroup$
  • $\begingroup$ Rate 1-5 is neither an up or down vote. You would penalize a reputation for a 1 (top) rating. $\endgroup$ – paparazzo Dec 16 '15 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.