1
$\begingroup$

I am building an app where I identify information from the SMS, something similar to expense management apps. I have a parser which reads all the SMS of user, identifies SMS of interest and parses useful information from it. This bit is working fine, however the problem is that I want to also score completeness and accuracy of the parser.

Completeness - Out of 100 SMS of interest, how many of them parser is able to identify
Accuracy - Out of SMS identified as relevant, for how many SMS parser is able to extract right information

I have a manual process to identify these. I take a random sample from the corpus along with parsing results and manually tag which ones are correct or wrong. I am able to identify gaps in parser and fix them by this process. However more gaps I fix more it becomes difficult to identify further gaps with this process. Also since sample is random, it has many SMS with same structure. Parsing success/failure for similar SMS is similar. Completeness and accuracy from the random sample does not correctly represents correctness and accuracy of whole corpus as well.

I was thinking that since financial SMS of interest have templates associated with them, if I can tag template of the SMS, I can keep track the templates which are known to be parsed correctly and only do manual checks on templates which are not yet tagged manually for correctness. This would also help me build tools to identify correctness, accuracy and unknown status for the whole corpus. It can also help me identify new templates as they are added to the corpus.

There are 1000s of templates. Examples of templates:

Template1 - Thanks for spending INR 100 at Amazon on 1st Jan, your available credit limit is 1,000
Template2 - You have spent INR 100 on your credit card ending in 1234 at Amazon on 1st Jan 2020
Template3 - You txn of INR 100 is credited/reversed by Amazon on 2nd Jan 2020, Avl Bal - 1,100
Template4 - You account xxxx1234 has been debited with INR 500 and bene A/C xxxx4321 has been credit on 3rd Jan

I have searched a lot on google and many results which seemed relevant were from this forum. Similarity based on n-gram seemed quite interesting, but I was not able to make much of how to apply it to my problem. One of the questions which seemed very similar: Identifying templates with parameters in text fragments. However, it differs slightly in the sense that it identifies sentences with similar templates out of given corpus. I want to tag template of a sentence without looking at the whole corpus. I am okay with training a model to tag the template, but I would want the model to identify new templates which are not part of original corpus and tag them as well.

I am a software engineer and have no prior experience with data science. Please pardon me if I am not making any sense or if this forum is not the right place for this question. I would really appreciate any help in identifying the right approach and some references to understand the concepts involved.

$\endgroup$
1
$\begingroup$

I'm not sure I understand your problem very well but let's see. First let me try to formalize the task as a ML problem:

  • Identifying the SMS of interest is a binary classification task. Your "completeness" score seems to correspond to the standard recall measure. It is usually a good idea to also look at precision, i.e. out of the SMS identified as relevant how many are truly relevant.
  • You don't really explain the second part of extracting information out of SMS of interest. In general this part might be some kind of sequence labelling problem (similar to Named Entity Recognition).

It's important to distinguish these two parts if you're going to use ML.

Now what you are currently doing is a rule-based system: you manually identify the patterns one by one, then the system just applies the list of patterns. rule-based systems are considered the ancestor of ML systems: as you noticed, it might be possible to automatize the part of identifying the patterns. Given that you already have labelled examples (training data), you could train a model to recognize the general patterns by itself. The main difficulty is to provide the model with the right features so that it can do the job as correctly as possible. Note that in general it doesn't really work as a list of templates in the same way as what you do manually.

The question of identifying cases without looking at the whole corpus might be related to ''semi-supervised learning'', where a system iteratively learns to annotate unlabelled data from an small initial set of labelled data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.