I am building an app where I identify information from the SMS, something similar to expense management apps. I have a parser which reads all the SMS of user, identifies SMS of interest and parses useful information from it. This bit is working fine, however the problem is that I want to also score completeness and accuracy of the parser.
Completeness - Out of 100 SMS of interest, how many of them parser is able to identify
Accuracy - Out of SMS identified as relevant, for how many SMS parser is able to extract right information
I have a manual process to identify these. I take a random sample from the corpus along with parsing results and manually tag which ones are correct or wrong. I am able to identify gaps in parser and fix them by this process. However more gaps I fix more it becomes difficult to identify further gaps with this process. Also since sample is random, it has many SMS with same structure. Parsing success/failure for similar SMS is similar. Completeness and accuracy from the random sample does not correctly represents correctness and accuracy of whole corpus as well.
I was thinking that since financial SMS of interest have templates associated with them, if I can tag template of the SMS, I can keep track the templates which are known to be parsed correctly and only do manual checks on templates which are not yet tagged manually for correctness. This would also help me build tools to identify correctness, accuracy and unknown status for the whole corpus. It can also help me identify new templates as they are added to the corpus.
There are 1000s of templates. Examples of templates:
Template1 - Thanks for spending INR 100 at Amazon on 1st Jan, your available credit limit is 1,000 Template2 - You have spent INR 100 on your credit card ending in 1234 at Amazon on 1st Jan 2020 Template3 - You txn of INR 100 is credited/reversed by Amazon on 2nd Jan 2020, Avl Bal - 1,100 Template4 - You account xxxx1234 has been debited with INR 500 and bene A/C xxxx4321 has been credit on 3rd Jan
I have searched a lot on google and many results which seemed relevant were from this forum. Similarity based on n-gram seemed quite interesting, but I was not able to make much of how to apply it to my problem. One of the questions which seemed very similar: Identifying templates with parameters in text fragments. However, it differs slightly in the sense that it identifies sentences with similar templates out of given corpus. I want to tag template of a sentence without looking at the whole corpus. I am okay with training a model to tag the template, but I would want the model to identify new templates which are not part of original corpus and tag them as well.
I am a software engineer and have no prior experience with data science. Please pardon me if I am not making any sense or if this forum is not the right place for this question. I would really appreciate any help in identifying the right approach and some references to understand the concepts involved.