I'm working in Python, using Scrapy, and NLTK to try to understand how I can extract data from college websites.
My scraper can navigate through the university websites and find their tuition fees pages perfectly , but when trying to extract specific fees like :
- Resident
- Non Resident
- Per Credit Hour
- Per Semester
I'm running into trouble due to the data being so unstructured from site to site.
I've tried using NLTK to parse data based on parts of speech tags and regex chunking to try to extract sentences such as "tuition cost for resident: $12,500" but colleges can display this data in a number of ways.
Here is my question:
Are there any better ideas/methodologies that I should be looking into that can help me with extracting this type of data?