0
$\begingroup$

I have data as below:

Carpanter
Carepnter
Carpentor
Labourer
Labor
Labour
Housewife
House Wife
housewife.

I want to clean data and rectify the spelling mistakes but not manually because its a huge data. Due to spelling mistakes these 50/60 occupations have become around 2000.

$\endgroup$
3
$\begingroup$

What you are trying to do is text normalization, which is a part of NLP pipeline.

To successfully preprocess your data, please try this below:

  1. Install ekphrasis

    pip install ekphrasis
    
  2. Apply seg & spell correction:

    from ekphrasis.classes.spellcorrect import SpellCorrector
    from ekphrasis.classes.segmenter import Segmenter
    sp = SpellCorrector(corpus="English")
    seg_eng = Segmenter(corpus="English") 
    
    words_to_correct = ['Carpanter','Carepnter','Carpentor','Labourer','Labor','Labour','Housewife','House Wife','housewife.']
    for word in words_to_correct:
        segmented = seg_eng.segment(word)
        corrected = sp.correct(segmented)
        print(word + " -> " + corrected)
    

Outputs as:

Carpanter -> carpenter
Carepnter -> carpenter
Carpentor -> carpenter
Labourer -> labourer
Labor -> labor
Labour -> labour
Housewife -> housewife
House Wife -> housewife
housewife. -> housewife

Note: Check this amazing work and paper if you are interested in!

| improve this answer | |
$\endgroup$
0
$\begingroup$

Using Pyenchant, this can be

import enchant
dictionary = enchant.Dict('en_US')
print(dictionary.suggest('word_to_be_checked'))
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.