0
$\begingroup$

I am trying to create a script to filter the most "intersting" articles from an rss feed and rank them.

feeds = ['http://feeds.theguardian.com/theguardian/technology/rss',
         'http://rss.cnn.com/rss/money_news_international.rss',
         'https://news.ycombinator.com/rss?format=xml',
         'http://feeds.reuters.com/reuters/companyNews',
         'http://feeds.reuters.com/reuters/businessNews']

I tried using applying the K-means algorithm to RSS feeds to filter the most popular articles from thousands of links in an attempts to reduce my personal RSS reading time.

However, I feel that this might not be state of the art.

Any suggestions on papers, actual implementations or approaches to get a proper list of "must-read" articles.

I appreciate your replies!

$\endgroup$
  • 1
    $\begingroup$ K-means clearly cannot capture popularity or interestingness... Commercial tools such as f...ly likely rely on tracked view counts by a large number of users, but you don't have the data for this. $\endgroup$ – Anony-Mousse Oct 6 at 5:50
1
$\begingroup$

Big Note: Unless you do not define what "interesting" is, Machine Learning can not do anything for you. Clustering only tells you which documents are similar to eachother but YOU need to define which cluster is interesting (BTW, clustering in terms of context is called Topic Modeling as each cluster of written text is about some topics).

Now my suggestions:

Topic Modeling

Use a simple LDA to detect topics of each document and cluster docuemnts based on topics. For each topic, print first 5 dominant keyword and if they interest you, then rad the articles.

Keyword Search

Set up a list of keywords you would like to read about. Using a simple TF-IDF model you can find "high score" docuemnts according to your favorite keywords.

Keyword Extraction

You may extract keywords of each document usng RAKE with a score higher than a manual threshold. See if keywords interest you or not. (conceptually close to LDA but here you can extract keywords from each individual document, while LDA makes sense if you have a corpus of text.)

Update

According to the additional info in the comments, the question turned even more towards classic recommender systems.

We have the number of clicks per article and we use it as the interestingness score. Now one may say (a very basic idea to be honest):

  1. Crawl all the articles and Use RAKE to extract keywords from n most interesting articles. Use those keywords as search queries.
  2. Use BM25 algorithm to search those queries and get bm25 score for each article according to queries.
  3. Aggregate those bm25 score with interestingness score (e.g. simply multiply them) and rank articles from most interesting to least interesting.
$\endgroup$
  • $\begingroup$ Thx for your answer! I thought about the term "interesting" and there are two solutions that I came up with, first "interesting" defined in as of clicks/engagement, so that sources/topics that users click might be more interesting that other - I guess that's how fb, google etc. are doing it - and the 2nd is that I would previously define based on keywords what I would like to filter for - such as how Google Alert does it currently. Any suggestions on further infor regarding "interesting" based on users clicks/engagement? Are there any code examples that one can study. I appreciate your reply! $\endgroup$ – Kare Oct 6 at 18:37
  • 1
    $\begingroup$ Your guess is pretty right! Click journeys are one of main sources of information in web analytics, user behavior modeling, recommender systems, etc. If you have number of clicks then you have a score for each document (simply the number of score or even better, number of clicks in a document divided by total number of clicks). Then you can sort docuements based their scores and do topic modeling on top ones. Or do topic modeling on all of them and then sum up the scores inside each cluster. Higher the sum is, more interesting is the topic. If it helped I apprecate considering "Accept ". $\endgroup$ – Kasra Manshaei Oct 6 at 19:31
  • 1
    $\begingroup$ For codes, tell me what u need and i redirect u to some resources. Cheers! $\endgroup$ – Kasra Manshaei Oct 6 at 19:31
  • $\begingroup$ Thx for your detailed reply! If you can add some code resources, about recommender systems and user behavior modeling this would be awesome! $\endgroup$ – Kare Oct 7 at 13:58
  • 1
    $\begingroup$ Welcome :) I am glad it could help! In this case user behavior is not much of interest because what each user does is not of your interest but what mass of users do (an article which is clicked by more users is better, regardless of each individual user). So it is turning to a kind of classic recommender system. I update the answer (not much about coding but try to be so explicit so you can code yourself) $\endgroup$ – Kasra Manshaei Oct 8 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.