# Text similarities

I have a list of E-mails, and I'm trying to find "similar" E-mails.

For example:

DanB@gmail.com

DanB1@gmail.com

Dan12B@yahoo.com

will be "similar" for me.

I'll be happy to hear if you are fimiliar with a method which can be applied here.

P.S. my available tools are Teradata (SQL) or R.

• Is this question about email addresses? Sep 21 '16 at 19:46
• Yes.. sorry if it wasnt clear Sep 21 '16 at 19:47

Since your question is super-huge and no one is going to do this job for you, here's a starting point for you to at least let you see how the various string comparison algorithms work:

library(stringdist)
library(stringi)
library(purrr)

emails <- c("ed@microsoft.com", "DanB@gmail.com", "DanB1@gmail.com", "Dan12B@yahoo.com",
"fred@outlook.com", "bill@juno.com", "Danb21@aol.com", "ed@tumblr.com")

naive_regex <- stri_match_all_regex(emails, "([[:print:]]+)@(.*)")

methods <- c("osa", "lv", "dl", "hamming", "lcs", "qgram", "cosine", "jaccard", "jw", "soundex")

map_df(methods, function(method) {
as.list() %>%
})
})


The output is too long to put here but it's a list of comparison matrices like this:

## [[2]]
## # A tibble: 10 × 9
##    looking_for  method    ed  DanB              DanB1            Dan12B  fred  bill            Danb21
##          <chr>   <chr> <chr> <chr>              <chr>             <chr> <chr> <chr>             <chr>
## 1         DanB     osa     4     0                  1                 2     4     4                 3
## 2         DanB      lv     4     0                  1                 2     4     4                 3
## 3         DanB      dl     4     0                  1                 2     4     4                 3
## 4         DanB hamming   Inf     0                Inf               Inf     4     4               Inf
## 5         DanB     lcs     6     0                  1                 2     8     8                 4
## 6         DanB   qgram     6     0                  1                 2     8     8                 4
## 7         DanB  cosine     1     0  0.105572809000084 0.183503419072274     1     1 0.387627564304205
## 8         DanB jaccard     1     0                0.2 0.333333333333333     1     1 0.571428571428571
## 9         DanB      jw     1     0 0.0666666666666668 0.111111111111111     1     1              0.25
## 10        DanB soundex     1     0                  0                 0     1     1                 0


Once you have an idea of acceptable tolerances, you can then create an algorithm to match => remove / match => remove, etc.

• I can't vote yet.. but your answer was very helpful.. thank you very much Sep 22 '16 at 5:25

You could try a very simple algorithm: https://en.wikipedia.org/wiki/Levenshtein_distance

https://cran.r-project.org/web/packages/stringdist/stringdist.pdf

After splitting the string by the '@'

• I can't vote yet.. but your answer was very helpful.. thank you very much Sep 22 '16 at 5:26