Since you were working with RegEx, I will ofer a RegEx solution.
I will also show that you need to also take care to first remove punctuation. (I will not go down the rabbit-hole of re-sinserting the punctuation back where it was!)
A RegEx solution:
import re
sentence = 'I need need to learn regex... regex from scratch!'
# remove punctuation
# the unicode flag makes it work for more letter types (non-ascii)
no_punc = re.sub(r'[^\w\s]', '', sentence, re.UNICODE)
print('No punctuation:', no_punc)
# remove duplicates
re_output = re.sub(r'\b(\w+)( \1\b)+', r'\1', no_punc)
print('No duplicates:', re_output)
Returns:
No punctuation: I need need to learn regex regex from scratch
No duplicates: I need to learn regex from scratch
\b
: matches word boundaries
\w
: any word character
\1
: replaces the matches with the second word found - the group in the second set of parentheses
The parts in parentheses are referred to as groups, and you can do things like name them and refer to them later in a regex. This pattern should recursively catch repeating words, so if there were 10 in a row, they get replaced with just the final occurence.
Have a look here for more detailed definitions of the regex patterns.
The more pythonic (looking) way
It has to be said that the groupby
method has a certain python-zen feel about it! Simple, easy to read, beautiful.
Here I just show another way of removing the punctuation, making use of the string
module, translating any punctuation characters into None (which removes them):
from itertools import groupby
import string
sentence = 'I need need to learn regex... regex from scratch!'
# Remove punctuation
sent_map = sentence.maketrans(dict.fromkeys(string.punctuation))
sent_clean = sentence.translate(sent_map)
print('Clean sentence:', sent_clean)
no_dupes = ([k for k, v in groupby(sent_clean.split())])
print('No duplicates:', no_dupes)
# Put the list back together into a sentence
groupby_output = ' '.join(no_dupes)
print('Final output:', groupby_output)
# At least for this toy example, the outputs are identical:
print('Identical output:', re_output == groupby_output)
Returns:
Clean sentence: I need need to learn regex regex from scratch
No duplicates: ['I', 'need', 'to', 'learn', 'regex', 'from', 'scratch']
Final output: I need to learn regex from scratch
Identical output: True
Benchmarks
Out of curiosity, I dumped the lines above into functions and ran a simple benchmark:
RegEx:
In [1]: %timeit remove_regex(sentence)
8.17 µs ± 88.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
groupby:
In [2]: %timeit remove_groupby(sentence)
5.89 µs ± 527 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
I had read that regex would be faster these days (using Python3.6) - but it seems that sticking to beautiful code pays off in this case!
Disclaimer: the example sentence was very short. This result might not scale to sentences with more/less repeated words and punctuation!