I have a data set in which users have input free text (think "APPLE", "Apple", "apple", "appl") and I'm trying to group the input by response type (e.g. organize all of the preceding examples by "Apple"). What's the best way to do this in Python?
2
-
1$\begingroup$ Define "by response type"? $\endgroup$– Sean OwenCommented Dec 21, 2015 at 22:46
-
$\begingroup$ Just updated the question to reflect what I'm looking for. Let me know if that clarifies things. $\endgroup$– user14910Commented Dec 22, 2015 at 0:03
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
That is called "entity resolution" or "record linkage". It is a very hard problem.
If it all possible, I would try to add "type ahead" to the form. That would encourage users to select already used categories.
fuzzywuzzy is a Python package for approximate string matching.
dedupe is "a Python library that uses machine learning to perform fuzzy matching, deduplication, and entity resolution quickly on structured data."
answered Jul 10, 2017 at 18:31