1
$\begingroup$

I have a big list of mail addresses (around one million) and I want to use/adapt/create an algorithm to find similarities between them (basically to cluster them).

All the algorithms I've checked so far e.g. n-grams variants, bag of words etc. are only used for clustering strings that are part of a context but here is not the case.

So far I've tried Levensthein, LCS. Both of them are pretty accurate but I'm interested in an algorithm that uses machine learning techniques, if such an algorithm exists for this task. Also they are pretty slow for my dataset.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ What current clustering did you use? And why do you say you want one that uses machine learning techniques? Wasn't your first technique already machine learning? $\endgroup$
    – Valentin Calomme
    Jul 12, 2018 at 11:46
  • $\begingroup$ @ValentinCalomme I want to see the differences between non ml and ml techniques, like, what are my options. I said I just checked those techniques (n-grams, bag of words), not that I implemented them. $\endgroup$
    – user3629077
    Jul 12, 2018 at 12:13
  • $\begingroup$ What labeled training data do you have? $\endgroup$
    – Has QUIT--Anony-Mousse
    Jul 12, 2018 at 17:04
  • $\begingroup$ nothing actually. I don't have a training data set. $\endgroup$
    – user3629077
    Jul 13, 2018 at 12:55

1 Answer 1

Reset to default
2
$\begingroup$

You said:

All the algorithms I've checked so far e.g. n-grams variants, bag of words etc. are only used for clustering strings that are part of a context but here is not the case.

However, those algorithms can still be used if you don't treat tokens as full words, but as combinations of characters instead.

Here is an example of such algorithm in Python using Sklearn.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

data = [
    '[email protected]',
    '[email protected]',
    '[email protected]',
    '[email protected]',
    '[email protected]'
]

X = TfidfVectorizer(analyzer='char').fit_transform(data)

kmeans = KMeans(n_clusters=2).fit(X)

for cluster in set(kmeans.labels_):
    print('\nCluster: {}'.format(cluster))
    for i, label in enumerate(kmeans.labels_):
        if label == cluster:
            print(data[i])

> Cluster: 0
> [email protected]
> [email protected]
> [email protected]
> 
> Cluster: 1
> [email protected]
> [email protected]

The major trick is TfidfVectorizer(analyzer='char'), where you don't treat n-grams of words, but of characters instead.

P.S. If you at some point want to use word n-grams (if you have generally well written sentences) you can use TfidfVectorizer(analyzer='word').

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.