Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have data as below:
Carpanter
Carepnter
Carpentor
Labourer
Labor
Labour
Housewife
House Wife
housewife.
I want to clean data and rectify the spelling mistakes but not manually because its a huge data. Due to spelling mistakes these 50/60 occupations have become around 2000.
What you are trying to do is text normalization, which is a part of NLP pipeline.
To successfully preprocess your data, please try this below:
Install ekphrasis
pip install ekphrasis
Apply seg & spell correction:
from ekphrasis.classes.spellcorrect import SpellCorrector
from ekphrasis.classes.segmenter import Segmenter
sp = SpellCorrector(corpus="English")
seg_eng = Segmenter(corpus="English")
words_to_correct = ['Carpanter','Carepnter','Carpentor','Labourer','Labor','Labour','Housewife','House Wife','housewife.']
for word in words_to_correct:
segmented = seg_eng.segment(word)
corrected = sp.correct(segmented)
print(word + " -> " + corrected)
Outputs as:
Carpanter -> carpenter
Carepnter -> carpenter
Carpentor -> carpenter
Labourer -> labourer
Labor -> labor
Labour -> labour
Housewife -> housewife
House Wife -> housewife
housewife. -> housewife
Note: Check this amazing work and paper if you are interested in!
Using Pyenchant, this can be
import enchant
dictionary = enchant.Dict('en_US')
print(dictionary.suggest('word_to_be_checked'))
