2
$\begingroup$

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man'

i tried this but this is slow

def split(sentence, word_to_index):
# words to index dictionary of valid words
    sentence_words = sentence.split()
    lst12=[] 
    for w in sentence_words:
            lst=[] 
            ws=w[:]
            # Set the (i,j)th entry of X_indices to the index of the correct word.
            while True:
                for wd in lst:
                    ws = ws.replace(wd,"")
                if ws=="":
                    lst=" ".join(lst)
                    lst12.append(lst)
                    break
                wrdind = ws[:]
                for kl in range(len(ws)):
                    try:
                        word_to_index[wrdind]
                        lst.append(wrdind)
                        break     

                    except KeyError:
                            wrdind = wrdind[:len(wrdind)-1]
                            continue
   return " ".join(lst12)  
```
$\endgroup$
3
$\begingroup$

This commonly called a "word break" problem. There are a variety of approaches, the most common use dynamic programming or tries. You can recursively try candidates and keep the candidates if they can split the entire string.

Here is a version (inspired by this answer):

from functools import lru_cache

@lru_cache(None)
def wordbreak(string):
    if len(string) < 1:
        return " "
    else:
        for i in range(len(string)):
            substring = string[:i+1]
            if substring in valid_words:
                remaining_split = wordbreak(string[i+1:])
                if len(remaining_split) > 0:
                    return substring + " " + remaining_split
                else:
                    return ""
            if (i == len(string) - 1) and (substring not in valid_words):
                return ""

valid_words = {'man', 'boycott', 'bears'}                                            
assert wordbreak('boycottbears') == 'boycott bears'
assert not wordbreak('abc')

However, creating a complete collection of valid words is impossible for Twitter data since there are always new words.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.