In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation is necessary, so we should assess the quality of the manually labeled data. It seems very expensive to have someone else check all the annotated data to assess the quality.
Then the question arises: how to assess the quality of the annotation?
From my experience, I learned that scholars may employ some tricks to filter out some low-quality surveys as described in this article: data quality checks. For instance, we can design a data set to be labeled(or corrected) with some amount of duplicated cases(or with some tiny changes that don't affect the labels) distributed randomly, and after the annotation, we can check if those cases are labeled consistently. Another example is that we can sample a small subset of the labeled cases and have some experts relabel them and estimate the quality of the whole dataset.
I wonder if there are some systematic methods to do such a data quality management task?