I have text infomration from several different sources. I need to identify the most reliable source in an unsupervized manner (no labels about true/false or ground truth to train on available). I think this field of work is sometimes referred to as latent truth discovery [in data integration systems].

The data looks like:

Row Source 1    Source 2    Source 3    ...     Source N
1   Bayes Inc.  Bayes Inc   Boys Inc.   ...     Bayes inc.
2   Mr. Foo     Mr Foo      Mr. Moon    ...     M. Foo
3   ACME Labs   Acme Lab    MathLab     ...     Giant Error

In the table above, Source 1 and 2 are (most) trustworthy while 3, N are not.

Problem: I want to identify the most trustworthy source (viz. column with lowest average error relative to others).

What I did: My current approach is to identify the "latent truth" by a kind of majority voting. For each row, pairwise Levenshtein distances (between columns/sources) are calculated. Finally, the column/source with the lowest mean of Levenshtein distances (over all rows) to other columns/sources is chosen as the "true" (most trustworthy) information.

There are several issues for now: First, the approach is "slow" since many Levenshtein distances need to be calculated. Second, the approach fails with very noisy data (low error rate or redundant additional text). Third, the solution will only work if errors are random and independent. However, there may be situations in which errors occur in a correlated way. Fourth, the mean (as an aggregated measure of error per pair) tends to be "high" if there are few large distances (viz. right skewed distribution of distances).

Question: What are possible alternative options to approach a problem like this?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.