I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations (so the cost of adding a word is the sum of the cost of adding each character)
I need to compute string similarity where different operations have different cost. The cost of modifying different substrings may be different The cost may depend on the location of the editing This cost is not explicit, but should be trainable from examples of pairs of similar and non similar strings
For example: for book name entity linking additional words may be the name of a series or the name of the author, words that are people names or words coming after "by" are more likely to be the author name
so "Tom Sawyer", "Tom Sawyer, Mark Twain", and "Tom Sawyer by Mark Twain" are equivalent but "Tom Sawyer by Mark Twain" and "Huckleberry Finn by Mark Twain" are different (even though the number of changed characters is similar)
preferably, I want to compute vector embedding of the strings in way that cosine similarity in the vector space would translate to string similarity under this "edit distance" (allowing for fast approximate nearest neighbours search)
My initial attempt is to use a transformer based neural network, but results aren't very good (yet).