5
$\begingroup$

I have a pandas dataframe with school names as one of the columns. However, there is quite a bit of misspelling in the school names, for example:

'Abernethy Elem School', 'Abernethy Elementary School', 'BOISE/ELIOT ELEM SCHOOL', 'Boise/Eliot Elementary School', 'Boise-Eliot Elementary School' ...

I am trying to clean up the names by doing this:

school_perf_report["SCHLNM"] = school_perf_report["SCHLNM"].str.lower().str.replace(r"elementary","elem").str.replace(r"/"," ").str.replace("-"," ").str.replace(r"\s+"," ")

Is there some cleaner way to perform the same operation? Basically

  1. Change "elementary" to "elem"
  2. Remove / or - and replace them with space.
  3. Remove multiple spaces
  4. Lower case everything

Thanks

$\endgroup$
  • $\begingroup$ What you can probably do is take that particular column, create a copy of it to be on safe side as another alias col, simply convert the newly created col to a list using .values, and then apply all the operations that you are supposed to do (in your case you have to use regex like you have shown above, re module, etc..) and then simply replace the original column and drop the alias too... Also what's the problem with your current version?? It seems fine to me..., Break it up if you don't like the length of the code.. $\endgroup$ – Aditya May 29 '18 at 19:37
  • $\begingroup$ There are a few more things I need to do, not just the 4 listed. This has resulted in a long chain of about 10 str.replace(), which looks ugly. The column has about 30k entries, so am wondering if list would be a good way. $\endgroup$ – user61034 May 29 '18 at 20:09
2
$\begingroup$

Create a definition for string replacements:

def change_string(x):
    return x.replace('|', '_').replace("elementary","elem").replace('old_value', 'new_value)

Then use map command to perform operation on entire column:

# You could create a new column to check whether the output is as expected
# If it is as expected, you could modify the original column
school_perf_report["SCHLNM_MODIFIED"] = school_perf_report["SCHLNM"].map(lambda x: change_string(x))

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.