3
$\begingroup$

I am searching for an automated method (ideally a python package) that produces a score to assess the credibility of a given text (e.g. from a webpage).

I am not searching for:

  • text complexity assessments (i.e. how long sentences are and how many difficult words are used) as for example flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall readability score, difficult words index, linsear write formula, or gunning fog.
  • text coherence (i.e. how well the next sentence fit with the previous one) as for example Text Coherence Analysis Based on Deep Neural Network

Why is complexity/coherence not the same credibility? Because many texts advertising for example homeopathy use long complex scientifically sounding and complex word loaded sentences while being nonsense in terms of trueness. Therefore I am wondering if there is any method to assess the credibility/reliability of a given piece of text/webpage information automatically?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I think you'll need to find a definition of "credibility" that can be expressed mathematically before anyone can really help you. I am not sure if a recommendation for a package is considered "on topic" in this forum either. $\endgroup$
    – Dylan
    Commented Jan 9, 2020 at 16:15

2 Answers 2

Reset to default
3
$\begingroup$

I don't think there's anything close to doing this:

  • It would be very hard to even define the task objectively, as different humans wouldn't agree about what is credible or not.
  • It would require a complex system to represent reliable background knowledge... and again people wouldn't agree what should be considered "reliable" or not.
  • Generally the state of the art in NLP is still far from solving tasks related to Natural Language Understanding in a satisfying way. Juding the credibility of a text requires not only a real understanding of the text but also an ability to reason at a higher level. It's not clear whether this level of AI can ever be reached.

If you find a package which pretends to achieve this task, try to apply it to its own documentation because it's not credible ;)

Improve this answer
$\endgroup$
1
  • $\begingroup$ I've found a nice recent paper ("Understanding and predicting Web content credibility using the Content Credibility Corpus" sciencedirect.com/science/article/pii/S0306457316306471) on this in the meantime and it seems you are right. Quote: "however, we still do not have algorithms that automatically evaluate Web content credibility to an extent that is adequately accurate and helpful to Web users. " $\endgroup$
    – lordy
    Commented Jan 10, 2020 at 9:35
2
$\begingroup$

The answer is extensively summarized in this recent 14-pages research paper "Veracity assessment of online data".

Main points:

  • "Three main veracity assessment research directions found, i.e., utilizing implicit features, employing explicit fact checking, and the appeal to authority method."
  • "The veracity assessment domain is still relatively immature."

EDIT: The above paper misses Credeye / Deepeye (https://gate.d5.mpi-inf.mpg.de/credeye/) which seems to be the only(?) method in this area that can be easily tested/used by other people.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.