regex to remove repeating words in a sentence

Question

I am new to regex. I am working on a project where i need to replace repeating words with that word. for example:

I need need to learn regex regex from scratch.

I need to change it to:

I need to learn regex from scratch.

I can identify the repeating words using the following regex \b(\w+)\b[\s\r\n]*(\l[\s\r\n])+

For substituting it, I need the word in the repeated word phrase. pattern.sub(sentence, <what do i write here?>)

Answer 1

Since you were working with RegEx, I will ofer a RegEx solution.

I will also show that you need to also take care to first remove punctuation. (I will not go down the rabbit-hole of re-sinserting the punctuation back where it was!)

A RegEx solution:

import re
sentence = 'I need need to learn regex... regex from scratch!'

# remove punctuation
# the unicode flag makes it work for more letter types (non-ascii)
no_punc = re.sub(r'[^\w\s]', '', sentence, re.UNICODE)
print('No punctuation:', no_punc)

# remove duplicates
re_output = re.sub(r'\b(\w+)( \1\b)+', r'\1', no_punc)
print('No duplicates:', re_output)

Returns:

No punctuation: I need need to learn regex regex from scratch
No duplicates: I need to learn regex from scratch

• \b : matches word boundaries
• \w : any word character
• \1 : replaces the matches with the second word found - the group in the second set of parentheses

The parts in parentheses are referred to as groups, and you can do things like name them and refer to them later in a regex. This pattern should recursively catch repeating words, so if there were 10 in a row, they get replaced with just the final occurence.

Have a look here for more detailed definitions of the regex patterns.

The more pythonic (looking) way

It has to be said that the groupby method has a certain python-zen feel about it! Simple, easy to read, beautiful.

Here I just show another way of removing the punctuation, making use of the string module, translating any punctuation characters into None (which removes them):

from itertools import groupby
import string

sentence = 'I need need to learn regex... regex from scratch!'

# Remove punctuation
sent_map = sentence.maketrans(dict.fromkeys(string.punctuation))
sent_clean = sentence.translate(sent_map)
print('Clean sentence:', sent_clean)

no_dupes = ([k for k, v in groupby(sent_clean.split())])
print('No duplicates:', no_dupes)

# Put the list back together into a sentence
groupby_output = ' '.join(no_dupes)
print('Final output:', groupby_output)

# At least for this toy example, the outputs are identical:
print('Identical output:', re_output == groupby_output)

Returns:

Clean sentence: I need need to learn regex regex from scratch
No duplicates: ['I', 'need', 'to', 'learn', 'regex', 'from', 'scratch']
Final output: I need to learn regex from scratch
Identical output: True

Benchmarks

Out of curiosity, I dumped the lines above into functions and ran a simple benchmark:

RegEx:

In [1]: %timeit remove_regex(sentence)
8.17 µs ± 88.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

groupby:

In [2]: %timeit remove_groupby(sentence)
5.89 µs ± 527 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

I had read that regex would be faster these days (using Python3.6) - but it seems that sticking to beautiful code pays off in this case!

Disclaimer: the example sentence was very short. This result might not scale to sentences with more/less repeated words and punctuation!

Answer 2

From what I understood, doing this with regex rules can be tricky and a bit slow. In python, I use this and it works perfectly :

from itertools import groupby
line_split = [k for k,v in groupby(line.split())]