I am trying to automatically categorize news articles according to their primary topics, i.e. politics, entertainment, sports, business, technology, health, etc.

There are some labeled datasets out there, but ideally I would like to create my own (for potential commercial usage later on). I am using python, but an answer clear enough with relation to any language would be sufficient.

So, what would be the best way to go about this task?

My current thoughts are:

  • Determine the most popular keywords for each category, then associate each keyword/keyword set with each respective category, and then use an algorithm to apply a "category" label to a large set of scraped articled based on the predefined keyword sets.

  • Another option is to scrape article from specific sections of news sites where the categories are already specified, and apply them to each individual constituent article.

After I have the trained dataset, I plan to implement Naive Bayes classification method to automatically categorize future articles.


As you can see I have some ideas, but because the web is a large and magical place, I assume someone with previous experience doing something like this may be able to reduce my effort expenditure by guiding me toward the most feasible solution.

  • $\begingroup$ I think the answer has something to do with Latent Dirichlet Allocation. Will update later when I can explain it. $\endgroup$
    – salamander
    Mar 1, 2018 at 15:22

1 Answer 1


Since news article categorization is a relatively common task, it would be fastest and easiest to use a already labeled training data.

Options include:

There is nothing preventing you from training a model on those datasets and then using that model for commercial purposes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.