I am working on the problem of automatic punctuation: given a stream of words, decide for each word whether there should be a punctuation mark after it (in future work I also want to distinguish between different punctuation marks, but currently it's a binary classification problem). My classifier uses a bidirectional LSTM whose output is fed into a multi-layer perceptron. The accuracy of the current classifier on the entire dataset is not good enough. In order to understand why, I want to create an "easier" dataset - a dataset on which even a simple (but not trivial) classifier would give good results. Then, I will make the dataset more and more difficult and see what improvements have to be made to the classifier. What is a good technique for creating synthetic datasets that are easier (but not trivial) to classify correctly?
In case it is relevant, here are some details on my system:
- Each word is represented by an embedding vector of length 200.
- The embedding vector is fed into a bidirectional LSTM with an output length of 50.
- The LSTM output is fed into a multi-layer perceptron with two layers, the hidden layer has size 60.
- The dataset has about 2000 paragraphs, each of which contains about 200 words.
- Since the classes are imbalanced (only 20% of the words are punctuated), I check the system accuracy using Cohen's kappa coefficient - a number between -1 and 1. A random classifier has kappa around 0. A good classifier has kappa above 0.7. My current classifier has kappa around 0.4 (average of 10-fold cross-validation), that is, better than random but not very good.
- I train the classifier in 80 iterations. After each iteration I check the kappa of the model on a validation set. Finally I return the model with the highest kappa.
