# Classification: How to manage data sets where one data row depends on another data row

I am trying to classify heading, image and image caption of a webpage. I am preparing data by scraping selected URLs(around 1000) using XPath of DOM elements I need. Each data row in the CSV file contains tag name, x-coordinate, y-coordinate, text-size etc with three target labels heading, image, image caption but a text simply will not become image caption until unless it is under an image. Is there any way to create this dependency among rows.

• What is the point of downvoting without telling why? Oct 30, 2017 at 11:58
• Use a linear chain CRF where the response (class) depends on the input and previous response. This is a type of sequential graphical model. The linked article also provides links to libraries. Welcome to the site and good luck!
– Emre
Nov 2, 2017 at 5:28
• @Emre waw this is amazing I did not know such thing exist. Thank you so much Nov 2, 2017 at 5:57

I assume that you are solving a supervised classification problem, that is, you train your model on a labeled sample. I can think of two approaches to this problem.

I. Classify tags, use neighbor tags for the features

For each tag you can calculate features like:

• (x, y) distances to the closest image
• (x, y) distances to the closest image above this tag
• number of images near this tag
• ...

Such features could be fed into any classifier like SVM or decision tree.

II Classify pairs of tags

For each tag $i$, you can consider all other tags $j$ and predict probability that $i$ is a caption of image $j$. This prediction may be based on concatenation of:

• features of $i$
• features of $j$
• joint features of $i$ and $j$, like vertical and horizontal distance from $i$ to $j$.

After these probabilities are predicted, you can classify $i$ as a caption indeed, if any of these probabilities exceeds some threshold.

• Thanks, man. I spent some time learning neural networks and I will try to solve this problem using RNN but your idea helped me to think in a different way Nov 9, 2017 at 4:29