0
$\begingroup$

This is intended to be mainly a reference request in the vast world of NLP and LLMs.

Context

A certain protocol is given in the form of text. This can be, for instance, the general description of a program in natural language or a statement regarding a legal or financial matter.

There is also a given set of basic protocols, which could be best-practices and/or basic algorithms for programming or basic regulations.

Goal

The protocol is supposed to implement or satisfy, at least partially, some of the basic protocols. The goal is to test the protocol against every basic protocol in order to get a binary classification:

1 - The protocol implements/satisfies (at least to some extent) the basic protocol.

0 - The protocol does not implement/satisfy (almost at all) the basic protocol.

Example

A very simple data science protocol could be the following.

Extract the data and check for completeness and consistency. Scale the numerical features, input the data types expected by data validation and load the dataset into the DW. Organize the scaling and data type inputting in two functions documented following Google style docstring.

Some basic protocols for this context and the related classification could be:

Basic Protocol Implemented
Check data accuracy False
Check data completeness True
Check data consistency True
Check data timeliness False
Check data validity False
Check data uniqueness False
Check data quality Probably True (Depends on "threshold")
Set expected data types True
Convert string features with less that 20% unique entries to category False
Follow numpy docstring False
Follow Google docstring True
Use a data catalog False

Problems

  1. The basic protocols could come from different sources and so have very different styles and potentially conflicting terminology and/or various degree of verbosity.
  2. Fine-tuning seems out of reach given the small amount of examples at hand.

Questions

Could you point me to some attempt to leverage LLMs in order to deal with similar problems?

I imagine that it may make sense to create some middle layer in which basic protocols are uniformized according to some well-chosen examples, to brake the protocol into smaller components and for the classification to possibly use a mixture of similarity measures and a majority voting given by a bunch of Yes/No questions given to the LLM.

For the case I am interested in, a simple vectorization of the protocol and basic protocols gave pretty poor results. I kind of expected it, given that all the basic protocols belong to the same context of the protocol.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.