0
$\begingroup$

I am looking for a way to calculate the string distance between two Pandas dataframe columns in a vectorized way. I tried distance and textdistance libraries but they require to use df.apply which is incredibly slow. Do you know any way to have a string distance using only column operations ?

Thanks

$\endgroup$
3
  • $\begingroup$ You could try using cosine similarity on a bag of words representation of the strings. $\endgroup$
    – Oxbowerce
    Feb 22, 2022 at 9:20
  • $\begingroup$ Thanks @Oxbowerce. Though, it's not ideal because I am measuring distances between email adresses, so the order of letters matters to me. $\endgroup$
    – Anatole
    Feb 22, 2022 at 9:47
  • $\begingroup$ If you use n-grams (e.g. sequences of 4 characters) instead of single characters for the bag of words representation you should still be able to take into account the order of the characters. $\endgroup$
    – Oxbowerce
    Feb 22, 2022 at 10:03

2 Answers 2

0
$\begingroup$

I have a similar problem and tried parallel computing using joblib. In terms of performance this approach seems okay. However, it appears that joblib "blocks" RAM memory when repeated very often. So I'm open for alternatives (or suggestions how to terminate the parallel job properly).

from joblib import Parallel, delayed
import distance
import pandas as pd
# https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html

# Define some distance measure
def calc_dist(myrow):
    return distance.levenshtein(myrow[0], myrow[1])

# Some fake data
df = pd.DataFrame({
     "text1":["some text","foo","bar","new text","more words"], 
     "text2":["text","hello","bar","bar","move words"]})

# Columns to lists / zip them 
l1=df['text1'].tolist()
l2=df['text2'].tolist()
nlist = list(zip(l1,l2))

# Calculate distances
dist_vec = Parallel(n_jobs=2)(delayed(calc_dist)(i) for i in nlist)

print(dist_vec)
> [5, 4, 0, 8, 1]
$\endgroup$
0
$\begingroup$

I found here that performance across string distance libraries varies greatly : https://github.com/life4/textdistance#benchmarks

The python-Levenshtein library is lightning fast compared to the others so I will use this one. If it's not sufficient I will use parallelism as suggested by @Peter

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.