2
$\begingroup$

I have a dataset of organization names that is quite messy. I used all the popular NER tools on it without much success(I assume it's because they lack context). I resolved to using OpenRefine but I reached a dead end with it's filters not picking up a lot of similar strings. I would like to use the data that I cleaned so far with OpenRefine for a (preferably supervised) machine learning algorithm that can afterwards continue the cleaning.
Are there any resources that could help with this?

$\endgroup$

2 Answers 2

3
$\begingroup$

Trifacta (https://www.trifacta.com/) supposedly can do that (learn from some examples provided by the user. I have no interest in the company, but it comes from academic research that I'm familiar with (http://vis.stanford.edu/wrangler/). Google had something free online (Google Refine) that can also do some intuitive things very simply, but I don't think it's as well developed. If you try either product, please let us know about your experience!

$\endgroup$
3
  • 2
    $\begingroup$ I also tried Trifacta at your suggestion. It seems to have a similar functionality to OpenRefine but with a prettier interface and a friendlier DSL. It also keeps a record of my steps but as a nice extra, it "translates" them into Python and I can export the script at the end and reuse it later. However, it didn't alter it's suggestions as a result of my usage as their service staff suggested. All in all, it's a nice tool (especially for non-programmers) and improves reproducibility/ reusability but I couldn't see any signs of learning. $\endgroup$ Sep 6, 2016 at 14:16
  • $\begingroup$ Google Refine is the old name of OpenRefine. $\endgroup$ Sep 6, 2016 at 14:39
  • 1
    $\begingroup$ Thanks for sharing your experience. It's very disappointing to know that Trifacta does not implement the learning component; it was one of their "big" calling cards. Let me add that 'text cleaning' is quite an 'ill-defined' field; there may not be a tool that does exactly what you need. My next suggestion would be to learn 'sed' and use the command line. $\endgroup$
    – Antonio
    Sep 7, 2016 at 16:59
0
$\begingroup$

I know this is a really old question, but if you can provide examples of the values that OpenRefine didn't cluster correctly, that you thought it should, we'd be happy to look at improving the tool.

Having said that, I'd probably approach your problem from a different angle and instead of using text similarity clustering, I'd attempt to reconcile against a database of organization names like Wikidata or OpenCorporates. This has the advantage that it should take into account things like aliases, former names, etc which aren't similar from a string similarity point of view, but which humans have curated. Both of the examples given have OpenRefine reconciliation services that can be used for this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.