I have a toy dataset of 10,000 strings of people's names, addresses and birthdays. As a quirk of the data collection process it is highly likely there are duplicate people caused by typos and I am trying to cluster them using K-means. I know there are easier ways of doing this, but the reason I am doing it like this is out of curiosity.

In order to vectorize each person I am concatenating the strings as follows: [name][address][birthday] and then running this through the following function to tokenise and clean the string:

def preprocess_text(text):
    # remove links
    text = re.sub(r"http\S+", "", text)
    # remove special chars and numbers
    text = re.sub("[^A-Za-z]+", " ", text)
    # remove stopwords
    if remove_stopwords:
        # 1. tokenize
        tokens = nltk.word_tokenize(text)
        # 2. check if stopword
        tokens = [w for w in tokens if not w.lower() in stopwords.words("english")]
        # 3. join back together
        text = " ".join(tokens)
    # return text in lower case and stripped of whitespaces
    text = text.lower().strip()
    return text

This outputs to a dataframe containing the corpus where each 'person' consists of string that is something like:

john smith belgrave road birmingham england

The entire corpus is then run through the TF-IDF vectorizer in sklearn.

When I use the Elbow method and the Silhouette Method to try and get the optimimal cluster number K for K-means, I get two graphs that don't show a clear value of K:

enter image description here

enter image description here

I've not really seen anything like this before and I was wondering if anyone has ideas for an optimal value of K based on these charts? Or, if these charts show that K-means really isn't a good way of doing this? Or, perhaps a TF-IDF vectorizer isn't a good approach and a Bag-of-Words vectoriser would be better as names and addresses should be semantically neutral? Any insight would be really appreciated. Thank you!


1 Answer 1


First, if you are interested in identifying duplicate records via clustering I would suggest you to use agglomerative hierarchical algorithm (e.g. single-linkage). It will result with a complete dendogramm and somewhere in the bottom of this dendogramm you will find your duplicated records.

Second, regarding TF-IDF representation, I am not sure it will yield a good result, since TF-IDF relies on word frequency, thus different spelling of the same name (e.g. Jim and James) or address will result in different vectors. For a somehow simplier solution I would run your data through any pre-trained language model (e.g. word2vec from gensim https://radimrehurek.com/gensim/models/word2vec.html), sum up all embeddings from the same person and then run either a KNN or a hierarchical clustering.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.