You should apply the same plan that you used to extract times to categorize those times:
Start with a rule-based system
Then try a machine learning approach
In order to build a machine learning, you'll need a collection of text labeled as "start-time", "end-time", or "neither". You can first try traditional algorithms like ...