Which (naive) NLP method for correlating human messages in chatrooms?

Question

Suppose an online chatroom is filled with many "alt" accounts - i.e, multiple accounts are being controlled by one user, a troll. This user leverages multiple accounts in order to steer the conversation in particular directions to suit their needs. This user may also be altering their mode of speech per account, to avoid detection.

Suppose I want to detect these alt accounts by using some kind of NLP classifier. What would the best method be? Where "best" means effective, but also relatively simple to set up - as in something you could do in python with relatively basic sklearn modules.

What I've tried so far is to collect the 100 most frequent words used by each user, and just throw that corpus into a sklearn.feature_extraction.text.TfidfVectorizer, and then looking at the pairwise similarity. These are mostly stop words, words which most NLP articles online are telling me to ignore. But I think these basic (almost unconsciously-used) words are less likely to be subject to obfuscation. The user may, for example, be applying varying spellings of less-common words (realize vs realise etc) - but the user could not plausibly have alternative spellings for ["the", "of", "you", "one"] etc.

Like I said, that isn't an "advanced" approach, but it's an example of the general level of complexity that I'm willing to undertake. Is there are any validity to that approach? If not, can you suggest a better one?

Answer 1

This is a stylometry task, more exactly some form of author verification/identification task. In case you want to dig deeper, the PAN workshop series is a good source of datasets and methods.

About your method:

• Your intuition is correct about selecting the most frequent words, in particular stop words: writing style is better characterized by the patterns in frequent grammatical constructs than by the choice of content words. However depending on the size of the data you might want to be more flexible about the number of words: if you have enough data, you should probably take more than the top 100 frequent words. If you don't, well... it might not work very well. Note also that some successful methods don't use the top frequent words but rather a range of middle frequency words.
• Using TFIDF is controversial for style identification. It's often done simply with the term frequency instead.

Other features you could consider:

• words n-grams, typically bigrams or even trigrams if there is enough data.
• characters n-grams (usually trigrams) have the surprising property to be very robust features for style detection. If you use these do not apply any tokenizer or remove punctuation signs.