5
$\begingroup$

I am performing an online news classification. The idea is to recognize group of news of the same topic. My algorithm has these steps:

1) I go through a group of feeds from news sites and I recognize news links.

2) For each new link, I extract the content using dragnet, and then I tokenize it.

3) I find the vector representation of all the old news and the last one using TfidfVectorizer from sklearn.

4) I find the nearest neighbor in my dataset computing euclidean distance from the last news vector representation and all the vector representations of the old news.

I have a problem using TfidfVectorizer because it weights more the special words that only appear in a few news, like Apple, and news that talk about Aple are grouped together even when they deal with different topics.

So, Is there a common approach more efficient than the one I am using?

| improve this question | |
$\endgroup$
2
$\begingroup$

The usual rule is to ask only one question per post. I will answer your first question.

Build a dictionary, once, in advance. With a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, and it's likely to contain all the words you need; the likelihood that you're missing some important word is pretty low. Now, each time you see an article, throw away any words not in the dictionary. That way, you won't need to vectorize the old news over and over every time; you can do it once and be done. That should improve performance.

Another variation: if you see a new word on some day, you can add that word to the dictionary; at the end of the day retokenize everything you saw that day.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Sorry about the multiple question. Throwing away any words not in the dictionary I can't recognize incoming classifications. For example, If in my dictionary it doesn't appear the words Apple and Jobs, I can't recognize beaking news related with Apple, which is a fresh topic. $\endgroup$ – Federico Caccia Apr 4 '18 at 18:16
  • 1
    $\begingroup$ @FedericoCaccia, with a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, and it's likely to contain all the words you need; the likelihood that you're missing some important word is pretty low. (Another variation: if you see a new word on some day, you can add that word to the dictionary; at the end of the day retokenize everything you saw that day.) I expect this will be good enough. $\endgroup$ – D.W. Apr 4 '18 at 18:49
  • $\begingroup$ @DW yes, the last recommendation is very useful! $\endgroup$ – Federico Caccia Apr 4 '18 at 19:31
2
$\begingroup$

We do something similar for financial news classification which I suspect is similar to what you're trying to do, the problem we hit when using completely automated classification was there are a number of rules you have to consider i.e. there are two Tesco's -the UK super market chain and a tractor company based in the states.

Instead, we built dictionaries based on Tf-Idf results which we would curate with rules i.e. Apple -pie.

Hope that helps

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.