I'm not sure if it meets all your criteria (mostly because I'm not sure I understand all your criteria!), but you could have a look at ELAN:
Description: With ELAN a user can add an unlimited number of textual
annotations to audio and/or video recordings. An annotation can be a
sentence, word or gloss, a comment, translation or a description of
There are many manual text annotation tools available, but you will probably have to search around in order to find the one which suits your precise needs.
Here are a few pointers:
Text annotation tools
a recent review
You ideally want a copy of The Handbook of Linguistic Annotation which covers the issues you’re up against in detail.
The basic idea is:
Create annotation guidelines as a training tool to increase interannotator agreement as far as possible
Measure interannotator agreement among people using your guidelines to get an idea of the irreducible error
Most machine learning algorithms are designed with complete trust in the labels. There is no standard way to model uncertainty in data labels. Thus, create a project-specific threshold for uncertainty to omit data or labelers. For example, a trusted classification data label would require n of m ensemble voting.
One major issue is re-labeling. Systems tend ...
So, I'm still not sure about what you consider to be an annotation on curlie.org. What I understood, you'll correct me if I'm wrong, is that you would like to annotate some text, more specifically you would like to annotate/identify concepts, and you would like to follow the structure on curlie.
From what I see, curlie doesn't contain any annotation, it's ...
It always depends on each specific problem. The amount of data depends on a number of factors including (but not limited to) the complexity of the problem, the number of features, the quality of the training data, the ratio of the training classes (i.e. class imbalance), etc.
If you have no idea where to start, sometimes the best thing is just to experiment....
Shannon entropy is a common measure of uncertainty among a fixed set of choices, as are the ones provided by Brian Spiering.
Regarding your question -- "some approach to compare named entities regarding how difficult to disambiguate?" -- note that the difficulty to disambiguate an entity is completely context and domain dependent. To give a truly useful ...
Entity Linking is a type of supervised machine learning, thus many of the common performance metrics could be used. In particular, creating a confusion matrix would identify where one label was predicted but the ground-truth was different. Confusion matrices can be calculated with counts or normalized, a normalized data would be an estimate of "ambiguity ...