I'm going to use ( RNN+Logisitic Regression ) to make sentiment analysis.
Should I do preprocessing for text like remove stop words, punctuation and extract keywords by found nouns ?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Welcome to the Data Science forum.
Yes, data preprocessing is an important aspect of sentiment analysis for better results. What sort of preprocessing to be done largely depends on the quality of your data. You'll have to explore your corpus to understand the types of variables, their functions, permissible values, and so on. Some formats including html and xml contain tags and other data structures that provide more metadata.
At a high level the sentiment analysis (using bag of words) will involve 4 steps:
Lets understand different possible data preprocessing activities:
Convert text to lowercase – This is to avoid distinguish between words simply on case.
Remove Number – Numbers may or may not be relevant to our analyses. Usually it does not carry any importance in sentiment analysis
Remove Punctuation – Punctuation can provide grammatical context which supports understanding. For bag of words based sentiment analysis punctuation does not add value.
Remove English stop words – Stop words are common words found in a language. Words like for, of, are etc are common stop words.
Remove Own stop words(if required) – Along with English stop words, we could instead or in addition remove our own stop words. The choice of own stop word might depend on the domain of discourse, and might not become apparent until we’ve done some analysis.
Strip white space – Eliminate extra white spaces.
Stemming – Transforms to root word. Stemming uses an algorithm that removes common word endings for English words, such as “es”, “ed” and “’s”. For example i.e., 1) “computer” & “computers” become “comput”
Lemmatisation – transform to dictionary base form i.e., “produce” & “produced” become “produce”
Sparse terms – We are often not interested in infrequent terms in our documents. Such “sparse” terms should be removed from the document term matrix.
To give you more insight onto the steps involved, here are some example sentiment analysis using logistic regressions codes https://github.com/srom/sentiment
Hope this helps.