I'm working on a project where I have a large dataframe of paintings (from the Art500k dataset), each row corresponds with a painting, containing the author's name in the author_name column. Sometimes, the same artist is stored under a different/incomplete name for two different pictures (rows), e.g. Rembrandt and Rembrandt van Rijn. I would like to "accurately" find the set of all aliases for each author, so I can store them in a dictionary and merge the "alias" data together.

The hard part is finding the aliases. Initially, I thought that I look whether some author names contain others, create a connection based on this information, and merge the connected "components" (graph notation) together. This would for example, merge Rembrandt, Rembrandt van Rijn, Rembrandt (Rembrandt van Rijn), and Rembrandt Harmensz. van Rijn together, which is correct. But it would also merge Rembrandt Peale with Rembrandt van Rijn, which is incorrect. I had further ideas for making connections based on a "similarity function", but such function would still find Rembrandt as similar to Rembrandt van Rijn, as to Rembrandt Peale, or less.
So now I consider using NLP, or any other method recommended for such cases, but I lack knowledge on this. Maybe I should gather nicknames of famous painters based on Wikipedia? What would be a good approach?

EDIT: I used different measures: fuzzywuzzy library for fuzzy string matching which returns a score for each pair of strings, and implemented some simple dataset-specific measures, then combined the measures with weights. Playing with weight parameters based on results are useful but there should be some other measure to help. I could imagine using LLMs or some other ML tools to help recognize.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.