Model for binary classification of links as "Article" or "Other"

Question

I'm creating a web crawler which must:

1. Fetch a web page.
2. Parse all <a> tags with hrefs on the page.
3. Classify the tags as either article (Meaning the link refers to a news article) or other (Refers to anything other than a news article).

I have created several working versions of this using:

• GPT-3
• Jurassic-1
• BigScience Bloom

However, I can't help but think that using these large language models might be overkill in terms of model size and certainly of expense. I know I can use smaller language models (GPT-J and Bloom variants). Are there any models besides large language models which might be better suited to this sort of classification task?

My intuition is that there might be but consider the enormous range of possible links and anchor texts to discern from.

Answer 1

I don't think you need to use language models. You can get a heur8stic model.

extract features: in first 75% of page find

• if there are comments ,
• ratio of number of words to number of images ,
• ratio of number of words to number of videos , and

make a training and validation dataset of 500 pages each. Use logistic regression.