2
$\begingroup$

I have a dataset of organization names that is quite messy. I used all the popular NER tools on it without much success(I assume it's because they lack context). I resolved to using OpenRefine but I reached a dead end with it's filters not picking up a lot of similar strings. I would like to use the data that I cleaned so far with OpenRefine for a (preferably supervised) machine learning algorithm that can afterwards continue the cleaning.
Are there any resources that could help with this?

$\endgroup$
3
$\begingroup$

Trifacta (https://www.trifacta.com/) supposedly can do that (learn from some examples provided by the user. I have no interest in the company, but it comes from academic research that I'm familiar with (http://vis.stanford.edu/wrangler/). Google had something free online (Google Refine) that can also do some intuitive things very simply, but I don't think it's as well developed. If you try either product, please let us know about your experience!

$\endgroup$
  • 2
    $\begingroup$ I also tried Trifacta at your suggestion. It seems to have a similar functionality to OpenRefine but with a prettier interface and a friendlier DSL. It also keeps a record of my steps but as a nice extra, it "translates" them into Python and I can export the script at the end and reuse it later. However, it didn't alter it's suggestions as a result of my usage as their service staff suggested. All in all, it's a nice tool (especially for non-programmers) and improves reproducibility/ reusability but I couldn't see any signs of learning. $\endgroup$ – Georgiana.b Sep 6 '16 at 14:16
  • $\begingroup$ Google Refine is the old name of OpenRefine. $\endgroup$ – Georgiana.b Sep 6 '16 at 14:39
  • 1
    $\begingroup$ Thanks for sharing your experience. It's very disappointing to know that Trifacta does not implement the learning component; it was one of their "big" calling cards. Let me add that 'text cleaning' is quite an 'ill-defined' field; there may not be a tool that does exactly what you need. My next suggestion would be to learn 'sed' and use the command line. $\endgroup$ – Antonio Sep 7 '16 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.