1
$\begingroup$

I am new to machine learning and AI, and I am working on a problem in which I need to clean a table (database) of text/words. For example I should delete words like and, the, etc. , or replace words like COMP with company or substitute other acronyms with the suitable words. I wanted to find some resources on the techniques proposed for this purpose and to find what else I can do to better clean the table. Thanks for your help.

$\endgroup$
3
  • 1
    $\begingroup$ You may do cleaning/substitution with grep-like tools in R, Python, or GNU Linux. No need in ML/AI tools. $\endgroup$ Commented Jan 25, 2016 at 21:23
  • $\begingroup$ @SergeyBushmanov thanks, I will try that. but I also want to understand the algorithms that are used for this purpose. $\endgroup$
    – Alex
    Commented Jan 25, 2016 at 21:32
  • $\begingroup$ Why do you need to clean the table? (By the way: Your table gets the cleanest with TRUNCATE TABLE ;-)) $\endgroup$ Commented Jan 26, 2016 at 14:15

1 Answer 1

1
$\begingroup$

For example I should delete words like and, the, etc. ...

In natural language processing, this first task is called stop word removal. You can identify them by looking at the words' frequency over the documents; uninformative words appear very often.

$\endgroup$
2
  • $\begingroup$ thanks, what other topics should I look into other than stop word removal? $\endgroup$
    – Alex
    Commented Jan 25, 2016 at 22:51
  • 2
    $\begingroup$ Look up "preprocessing". Tokenization, stemming, lemmatization. $\endgroup$
    – Emre
    Commented Jan 25, 2016 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.