I am trying to classify heading, image and image caption of a webpage. I am preparing data by scraping selected URLs(around 1000) using XPath of DOM elements I need. Each data row in the CSV file contains tag name, x-coordinate, y-coordinate, text-size etc with three target labels heading, image, image caption but a text simply will not become image caption until unless it is under an image. Is there any way to create this dependency among rows.

  • $\begingroup$ What is the point of downvoting without telling why? $\endgroup$
    – Praveena
    Commented Oct 30, 2017 at 11:58
  • $\begingroup$ Use a linear chain CRF where the response (class) depends on the input and previous response. This is a type of sequential graphical model. The linked article also provides links to libraries. Welcome to the site and good luck! $\endgroup$
    – Emre
    Commented Nov 2, 2017 at 5:28
  • $\begingroup$ @Emre waw this is amazing I did not know such thing exist. Thank you so much $\endgroup$
    – Praveena
    Commented Nov 2, 2017 at 5:57

1 Answer 1


I assume that you are solving a supervised classification problem, that is, you train your model on a labeled sample. I can think of two approaches to this problem.

I. Classify tags, use neighbor tags for the features

For each tag you can calculate features like:

  • (x, y) distances to the closest image
  • (x, y) distances to the closest image above this tag
  • number of images near this tag
  • ...

Such features could be fed into any classifier like SVM or decision tree.

II Classify pairs of tags

For each tag $i$, you can consider all other tags $j$ and predict probability that $i$ is a caption of image $j$. This prediction may be based on concatenation of:

  • features of $i$
  • features of $j$
  • joint features of $i$ and $j$, like vertical and horizontal distance from $i$ to $j$.

After these probabilities are predicted, you can classify $i$ as a caption indeed, if any of these probabilities exceeds some threshold.

  • $\begingroup$ Thanks, man. I spent some time learning neural networks and I will try to solve this problem using RNN but your idea helped me to think in a different way $\endgroup$
    – Praveena
    Commented Nov 9, 2017 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.