I am building a project for my bachelor thesis and am wondering how to prepare my raw data. The goal is to program some kind of semantic search for job postings. My data set consists of stored web pages in HTML format, each containing the detail page of a job posting. Via an interface I want to fill in predefined fields like skills, highest qualification, etc. with comma-separated sentences or words. These are then embedded via a Hugging Face Transformer and afterwards the similarity of the input is to be compared with the already embedded job postings and the "best match" is returned.
I have already found that intensive preprocessing such as stop word removal and lemmatization is not necessarily required for transformers. However, the data should be processed to resemble the data on which the pre-trained transformers learned. What would be the best way to prepare such a data set to fine-tune pre-trained Hugging Face Transformers?
Additional info: 55,000 of the saved web pages contain an annotation scheme via which I could simply extract the respective sections "Skills" etc. from the HTML text. If that is not sufficient, I can use prodigy to further annotate the data, e.g. by span labeling texts within the text of the job postings.
Thank you very much in advance!