2
$\begingroup$

I'm working on a project that requires determining if the page representing a hosted file on a third party platform (such as rapidgator or nitroflare) is still up or not.

For example, here is a file that is still up.

Here is a page representing a file that is no longer up.

I have identified several hundred file hosting services and each of them have their own way of displaying up vs down pages.

Is it possible to design/train a model that would classify screenshots of these hosted file pages as up vs down? If so what approach is recommended? Does using a method other than image classification make more sense here?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
0
$\begingroup$

Yes, instead of image classification you should look to scrape the websites themselves to get to file links or down messages directly.

You could still use an ML to determine down messages with minimal user input but it will be much faster and easier than image recognition.

Image recognition will fail you here because to start you have a population of 100 or so which is a small sample and you'd need to hand label at least 50 to train any kind of viable classifier at which point you might ad well just hand label them all.

Improve this answer
$\endgroup$
1
  • $\begingroup$ There are several problems with the scraping approach: 1) DOM structure changes periodically, 2) the wording used to indicate up/down pages changes periodically, 3) links are obfuscated, and 4) site specific scrapers need to be maintained when new sites come up. It's certainly doable but my original question was meant to explore other possibilities that require less maintenance. $\endgroup$
    – novon
    Oct 11, 2019 at 18:57
0
$\begingroup$

It is possible, but here is my take on this. You will most likely find it difficult to use it on new data. I don't feel like your network will be able to generalize that well since you will most likely lack training data, even if you have new website and label I don't think online learning for this case(given your model is only trained on very few samples) will work.

Indeed there must be a better way to do this. Maybe if you have seen a page that post several mirrors, and they have indicator that says whether the link is up/down and this is 100 percent correct. This I believe shows that there must exist a tool that allows such interaction

Improve this answer
$\endgroup$
0
$\begingroup$

Probably treating this problem as a text classification one would generalize in a better way than using screenshots. Most „down“ pages will likely say something indicating that the service is no longer available. So scrapping the visible text of the pages and using a simple count vectorizer or so could be more reliable than classifying images.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.