2
$\begingroup$

Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and reformatting data from a particular source as it is the only way from my point of view.

Exmaple:

Source 1: New Great Sneakers NGS1
Source 2: ngs1 sneakers

So it should be merged after applying rules: convert to lower case -> find some pattern 'NGS*' -> be sure that a number after pattern 'NGS' is the same.

It is some kind of naive way to solve the problem. Is there a better way to perform such task on regular basis?

$\endgroup$

2 Answers 2

0
$\begingroup$

That problem is commonly called record linkage.

There are many approaches and software options for record linkage. In your example, it can be done with conditional logic and regular expressions.

$\endgroup$
0
$\begingroup$

When you deal with similar strings which need to be merged, you can look at string distance metrics.

There are a number of packages/libraries, e.g for R or Python, which deal with merging similar strings. However, I think there is no off the shelf solution which does fuzzy string matching or record linkage in a (reliable) fully automatic way.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.