31
$\begingroup$

Given a sentence like:

Complimentary gym access for two for the length of stay ($12 value per person per day)

What general approach can I take to identify the word gym or gym access?

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
32
$\begingroup$

Shallow Natural Language Processing technique can be used to extract concepts from sentence.

-------------------------------------------

Shallow NLP technique steps:

  1. Convert the sentence to lowercase

  2. Remove stopwords (these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words)

  3. Extract n-gram i.e., a contiguous sequence of n items from a given sequence of text (simply increasing n, model can be used to store more context)

  4. Assign a syntactic label (noun, verb etc.)

  5. Knowledge extraction from text through semantic/syntactic analysis approach i.e., try to retain words that hold higher weight in a sentence like Noun/Verb

-------------------------------------------

Lets examine the results of applying the above steps to your given sentence Complimentary gym access for two for the length of stay ($12 value per person per day).

1-gram Results: gym, access, length, stay, value, person, day

Summary of step 1 through 4 of shallow NLP:

1-gram          PoS_Tag   Stopword (Yes/No)?    PoS Tag Description
-------------------------------------------------------------------    
Complimentary   NNP                             Proper noun, singular
gym             NN                              Noun, singular or mass
access          NN                              Noun, singular or mass
for             IN         Yes                  Preposition or subordinating conjunction
two             CD                              Cardinal number
for             IN         Yes                  Preposition or subordinating conjunction
the             DT         Yes                  Determiner
length          NN                              Noun, singular or mass
of              IN         Yes                  Preposition or subordinating conjunction
stay            NN                              Noun, singular or mass
($12            CD                              Cardinal number
value           NN                              Noun, singular or mass
per             IN                              Preposition or subordinating conjunction
person          NN                              Noun, singular or mass
per             IN                              Preposition or subordinating conjunction
day)            NN                              Noun, singular or mass

Step 4: Retaining only the Noun/Verbs we end up with gym, access, length, stay, value, person, day

Lets increase n to store more context and remove stopwords.

2-gram Results: complimentary gym, gym access, length stay, stay value

Summary of step 1 through 4 of shallow NLP:

2-gram              Pos Tag
---------------------------
access two          NN CD
complimentary gym   NNP NN
gym access          NN NN
length stay         NN NN
per day             IN NN
per person          IN NN
person per          NN IN
stay value          NN NN
two length          CD NN
value per           NN IN

Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym, gym access, length stay, stay value

3-gram Results: complimentary gym access, length stay value, person per day

Summary of step 1 through 4 of shallow NLP:

3-gram                      Pos Tag
-------------------------------------
access two length           NN CD NN
complimentary gym access    NNP NN NN
gym access two              NN NN CD
length stay value           NN NN NN
per person per              IN NN IN
person per day              NN IN NN
stay value per              NN NN IN
two length stay             CD NN NN
value per person            NN IN NN


Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym access, length stay value, person per day

Things to remember:

  • Refer the Penn tree bank to understand PoS tag description
  • Depending on your data and the business context you can decide the n value to extract n-grams from sentence
  • Adding domain specific stop words would increase the quality of concept/theme extraction
  • Deep NLP technique will give better results i.e., rather than n-gram, detect relationships within the sentences and represent/express as complex construction to retain the context. For additional info, see this

Tools:

You can consider using OpenNLP / StanfordNLP for Part of Speech tagging. Most of the programming language have supporting library for OpenNLP/StanfordNLP. You can choose the language based on your comfort. Below is the sample R code I used for PoS tagging.

Sample R code:

Sys.setenv(JAVA_HOME='C:\\Program Files\\Java\\jre7') # for 32-bit version
library(rJava)
require("openNLP")
require("NLP")

s <- paste("Complimentary gym access for two for the length of stay $12 value per person per day")

tagPOS <-  function(x, ...) {
  s <- as.String(x)
    word_token_annotator <- Maxent_Word_Token_Annotator()
    a2 <- Annotation(1L, "sentence", 1L, nchar(s))
    a2 <- annotate(s, word_token_annotator, a2)
    a3 <- annotate(s, Maxent_POS_Tag_Annotator(), a2)
    a3w <- a3[a3$type == "word"]
    POStags <- unlist(lapply(a3w$features, `[[`, "POS"))
    POStagged <- paste(sprintf("%s/%s", s[a3w], POStags), collapse = " ")
    list(POStagged = POStagged, POStags = POStags)
  }
  
  tagged_str <-  tagPOS(s)
  tagged_str

#$POStagged
#[1] "Complimentary/NNP gym/NN access/NN for/IN two/CD for/IN the/DT length/NN of/IN stay/NN $/$ 12/CD value/NN per/IN     person/NN per/IN day/NN"
#
#$POStags
#[1] "NNP" "NN"  "NN"  "IN"  "CD"  "IN"  "DT"  "NN"  "IN"  "NN"  "$"   "CD" 
#[13] "NN"  "IN"  "NN"  "IN"  "NN"

Additional readings on Shallow & Deep NLP:

Improve this answer
$\endgroup$
4
  • $\begingroup$ Excellent answer (+1). Just one suggestion: if possible, provide literature or, at least, general references for the shallow NLP technique that you've mentioned. $\endgroup$
    – Aleksandr Blekh
    Mar 25, 2015 at 13:17
  • 1
    $\begingroup$ Thank you so much. Two questions, can I do this with nltk? Could I use tf-idf to do the same, then take the most unique words (highest scores) as my key words? $\endgroup$
    – William Falcon
    Mar 25, 2015 at 13:30
  • $\begingroup$ @ Aleksandr Blekh, thanks. I have added additional reading links for learning more about shallow and deep NLP. Hope this helps $\endgroup$
    – Manohar Swamynathan
    Mar 25, 2015 at 13:56
  • $\begingroup$ @ William Falcon, thanks. 1) Yes, you can use nltk 2) Absolutely, TF-IDF can be used If you are trying to find the concept or theme at document(s) level. $\endgroup$
    – Manohar Swamynathan
    Mar 25, 2015 at 13:59
3
$\begingroup$

You need to analyze sentence structure and extract corresponding syntactic categories of interest (in this case, I think it would be noun phrase, which is a phrasal category). For details, see corresponding Wikipedia article and "Analyzing Sentence Structure" chapter of NLTK book.

In regard to available software tools for implementing the above-mentioned approach and beyond, I would suggest to consider either NLTK (if you prefer Python), or StanfordNLP software (if you prefer Java). For many other NLP frameworks, libraries and programming various languages support, see corresponding (NLP) sections in this excellent curated list.

Improve this answer
$\endgroup$
0
$\begingroup$

If you're a R user, there is a lot of good practical information here. Look at their text mining examples.
Also, take a look at the tm package.
This is also a good aggregation site.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Links aren't considered valid answers on this site. Please answer the original question in your post and use links to supplement your answer. $\endgroup$
    – sheldonkreger
    Mar 23, 2015 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.