7
$\begingroup$

I am looking to compile a sentiment corpus for news articles in multiple languages (~100k per lang. for a machine learning experiment) where each article is labeled positive, neutral, or negative. I have searched high and low but could not find anything like this available. I already have the news articles in each language.

My question to the community is how would you achieve this as accurately as possible?

I was first looking at Mechanical Turk, where you can hire people to label each article manually for you. And this may be the best way forward but expensive. enter image description here

Next, I thought about all of those existing popular libraries (some of whom have already used Mechanical Turk) that do sentiment analysis (AFINN, Bing Liu, MPQA, VADER, TextBlob, etc.)

  1. Sentiment Idea

My current idea is that I run each news article across a few of these libraries (for example AFINN, then TextBlob, then VADER) and for those articles who show positive, negative, neutral unanimously though all three libs are accepted into the corpus. Does that seem like a fairly strong and reasonable verification process?

  1. Language Idea

The next issue pertains to language itself. The 3 lib pipeline above can be executed in English with no issue. However these libraries do not uniformity support many other languages (Spanish, German, Chinese, Arabic, French, Portuguese, etc.) I was thinking about doing what VADER suggests and taking the news stories in non-english languages and sending them though Google Translation API to get them into English and then send them through the existing 3 lib pipeline above. I do realize there will be a loss in semantics for many articles. However, my hope is that enough articles will translate well enough that some pass through the 3 lib pipeline.

I am aware that translating and sending news articles through this triple blind sentiment pipe may take a 100k corpus and yield 10k results. I am fine with that. The accuracy and then price is my concern. I can easily acquire more data.

What would you do that may be a more accurate way of achieving a sentiment corpus of news articles? Is there an existing best used practice for assembling a corpus like this?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ I think the main problem with this is that the longer an article the less clear the sentiment. A long news article can easily contain positive, negative and neutral parts in different ratios. That's why you see sentiment analysis mostly for short text like survey responses, tweets, etc. Is it pretty clear from your articles what sentiment they should be assigned to? $\endgroup$
    – oW_ ♦
    Nov 20, 2018 at 18:59
  • $\begingroup$ Interesting. Thank you for your response! It is not clear what the overall sentiment is for the files. $\endgroup$
    – Chris
    Nov 20, 2018 at 19:42
  • $\begingroup$ @Chris if it is not clear what the sentiment is, why do you want to label them then? What is the goal of having the sentiment per article? $\endgroup$
    – Bruno Lubascher
    Nov 22, 2018 at 19:27
  • $\begingroup$ Thank you @BrunoGL. I would like to label each article for a training set. These are random articles. I am using them specifically for generalization (training a NN) across the corpus. $\endgroup$
    – Chris
    Nov 23, 2018 at 18:59
  • $\begingroup$ How about labeling not the whole article with 1 single sentiment but rather, each lines in the articles can have their own sentiment? $\endgroup$
    – atmarges
    Nov 25, 2018 at 16:50

1 Answer 1

Reset to default
1
$\begingroup$

Several questions and thoughts come to mind.

  1. What languages are in the corpus? This may impact what services you can leverage.
  2. I like the "Sentiment Idea" for languages that are supported natively by the services you mentioned.
  3. I would keep the "Language Idea" as the last resort as it is possible that the translation engine may not capture the sentiment of the original language.
  4. Mechanical Turk would be a good option if you can limit the number of samples sent for classification. For each language, you could try clustering the passages by, for example, word count into 30 (you pick) clusters and then perform sampling within the clusters to identify candidate passages to send to Mechanical Turk. I have used this technique to try to sample across the vector space more uniformly.

Don't dismiss oW_'s comment. You should seriously consider breaking the articles into paragraphs. You can always aggregate the paragraph scores to the article, but it's hard to get one representative score as the text gets longer.

HTH

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.