I have a data set in which the data is coming from various sources. Approx 3k records were verified manually and respective source is tagged if the data comes from that source and is valid/correct.

I am trying to implement a Voting Classifier for this scenario. The basic flow of which is shown below:

basic flow

Refer a sample of the accuracy and weight table below:

Sources Accuracy Coverage
src_1 0.62 0.90
src_2 0.89 0.85
src_3 0.80 0.10
so on... ... ...

I am directly taking the accuracies as the weights. However, I feel that coverage too has a role to play here. I am unable to figure out a way to formulate the combination of accuracy along with coverage.

For example w.r.t src_3 in the Sources column, the accuracy doesn't seem like a justifiable weight for the source considering the coverage on the Data Set.

How do I go about it?

  • $\begingroup$ Welcome to DataScienceSE. Is there a way for the meta-system to know whether a particular instance is covered by a particular source? If yes the sources could be selected based on whether they cover the input instance, and this way you don't need to take coverage into account anymore. For example source 3 would be selected less often but when it's selected its accuracy score is valid. Of couse a more simple method would be to take the mean of accuracy and coverage, for instance. $\endgroup$
    – Erwan
    Nov 22 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.