5
$\begingroup$

I have a dependency treebank including 100 sentences, which I divide into a training set and a test set. I extract some rules ((DS,PS) pairs) to convert the treebank to phrase structures. When I extract such rules from the training set, I can measure the percentage of rules (DS patterns) that cover the test set, suppose

(10, 24%), (20, 34%), (30,40%), (40,44%), (50, 55%),(60, 58%), (70, 61%)...

As you see as I increase the size of the training set, the coverage of extracted patterns increases! however its not linear!, I want to see how many data I need to reach 100% coverage? I guess I can use a regression, but which regression? logarithmic?

Is this related to 'learning curve'? if yes how can I use regression for a learning curve?

$\endgroup$
  • $\begingroup$ You do not specify your training/test split. Given that only have 100 total data points, you are most likely going to overfit the training data with a model of significant complexity. $\endgroup$ – Brian Spiering Jul 26 '17 at 16:41
1
$\begingroup$

You can never get 100% coverage for real-world grammar extraction. Grammar is complex and undefined for real-world data. In addition, it is an "open world" data problem because novel grammar phrases can always be created.

You might get 100% coverage for a small data set through overfitting.

Statistical learning theory provides a framework for predicting the limits of machine learning.

|improve this answer|||||
$\endgroup$
0
$\begingroup$

There is no general relationship between size of train dataset and accuracy or coverage on test dataset. To see this in a trivial way, you can augment your training data with unlimited copies of a single training example but this will (probably) not help your accuracy or coverage on the test dataset.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.