2
$\begingroup$

I would like to write an algorithm to convert unstructured texts (with contest descriptions) to structured data with the following fields:

  • contest start date (optional)
  • contest end date
  • main prize
  • additional prizes (optional)

I have hundreds of text examples, that could be used for model learning.

How to approach this task? Just in case this is important - the preferable language is Python. But I never worked on such tasks before. What is the best practice to do that?

$\endgroup$
5
$\begingroup$

If I get you right, you would like to extract the mentioned entities from the unstructured text. This sounds as Named Entity Recognition (NER) problem for me. In your case I would define the following entities:

  • START_DATE
  • END_DATE
  • MAIN_PRIZE
  • EXTRA_PRIZE

Your text now can be tagged or annotated with those entities. You would like to build a machine which will automatically tag the encountered entities in the unstructured text (free text).

I would suggest to take a look at Conditional Random Fields (CRF), a classic approach to NER. There are also Recurrent Neural Networks (RNN) which are state of the art for sequence modeling but I am not an expert with those methods.

Let's begin the process of model building. You need to decide what will be a single training example. This can be a sentence and this can be also a paragraph.

Since you did not provide any example, let me invent two:

The contest started on <START_DATE>july 1st</START_DATE> and ended on <END_DATE>july 25</END_DATE>.

The winner got away with <MAIN_PRIZE>1 million dollars</MAIN_PRIZE> and <EXTRA_PRIZE>bunch of flowers</EXTRA_PRIZE>.

Notice the way how the text is structured. Every word now is either label (surrounded by tags) or not labeled at all (outside of any tag). This is one way how to structure the text. This would be also the desired output on some unstructured text.

The machine does not understand the text and annotation. You need to come up with features that you can extract from each word. It can be the word itself, the previous word, the next word, left bigram, right bigram, suffix of length 1,2,3, prefixes of length 1,2,3 and so on. I suggest you to look at Stanford NER features.

Once you got the features, you need to create a training set. The training set is your annotated text (see my example) converted to label + feature information. It may look like this

O   feature1 feature2 ...
O   feature1 feature2 ...
B-START_DATE   feature1 feature2 ...
I-START_DATE   feature1 feature3 ...
...

This basically says that the first two words are not labeled, but the last two are part of an entity START_DATE.

Each line is essentially a word followed by features for it. The label "O" is the way to say that the word is not labeled, the labels B- and I- for DATE mean the first and second words of a named entity (e.g. "july 1st").

The sentences can be separated by empty lines.

Once you've got the training set file ready, you can do the model training. With a model in hand, you can apply it on any unstructured data (extracted features) to get the desired entities extracted.

As a tool I suggest you CRFSuite (also available with Python bindings). I can detail more each step if you would like.

$\endgroup$
1
$\begingroup$

Look for NLTK, a Python package for analysing natural language data. With it, you can add some structure to text as written by humans.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.