Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and reformatting data from a particular source as it is the only way from my point of view.
Source 1: New Great Sneakers NGS1 Source 2: ngs1 sneakers
So it should be merged after applying rules:
convert to lower case -> find some pattern 'NGS*' -> be sure that a number after pattern 'NGS' is the same.
It is some kind of naive way to solve the problem. Is there a better way to perform such task on regular basis?