# Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code.

• The fields we want are such as product title, price, quantity and shipment
• The generic function will be required to work on every kind of HTML pages containing relevant contents. This is big challenge of this task.

My attempts:

• I try to use nltk build-in method to do the information extraction, following a standard process: raw text --- sentence segmentation --- word tokenization --- part of speech tagging --- named entity extraction. I take the entire page of HTML text as the raw data. I find 1) the POS tagging can make mistakes and 2) the regex grammar is hard to set for a generalized case of each field. The difficulty comes from the variety of components in the NP phrase and also uncertainty of the start and end of NP phrases.
The example below is the python code for finding price, quantity etc.
import nltk
from nltk.tree import Tree
grammar = r"""
NP:
{<NN|NNP><NNP>?<CD|VBN>}          # Chunk everything
}<VBD|JJ>+{      # Chink sequences of VBD and JJ
"""
cp = nltk.RegexpParser(grammar)
cp.parse(sent)


The corresponding text from my targeted HTML code is

"Total Purchase Price : USD 1007.15"
"Quantity left: Unlimited"


It becomes low efficiency in developing a generic code to extract price/quantity value in each possible case (even worse than just merely using regular expression to grab the keyword and the potential value immediately after it). Moreover, some desired filed hided in the attribute rather than free text between tags. For example, the product titles in my targeted HTML page are set up as attribute title of tag <img>.

• I also try to think of building the HTML tag trees and consider if I can identify the main block of the information in a HTML page and reduce the searching scope on the HTML source code. As what I can find from HTML, the page code can be highly flexible and no rule one can rely on to parse HTML along a fixed way.

So, I need suggestion: what open source tool I can use to efficiently conduct this task?