0
$\begingroup$

I have a document classification problem where I need to classify whether a certain document is about real estate or not. I get a URL of a webpage from which I extract all the text and then using my trained model which is actually LSTM based I classify whether it is about real estate property or not. Here a page with real estate property means that the page should be talking about only one property and not more.

My model could get one of the following kinds of input data:

  1. A URL having a unique real estate listing and talking about that specific property. e.g. 1
  2. URL containing a list of real estates fulfilling a certain criteria. e.g. all the properties having 3 bedrooms, within a specific rent range. e.g. 2. These kind of pages I refer to as index pages.
  3. Just some random URL from these broker websites talking about their organization, their achievements, their teams, etc. e.g. 3

Different websites show a collection or lists of real estates differently than others. Some broker websites might have a list of real estates with each real estate as a hyperlink and a text (usually title of the real estate). In scenario 2, a list item might have few more details in addition to the title of the real estate. The former classifies my model as not a real estate but the latter kind of webpages confuses and the model has a tendency to classify them as real estate. In scenario 3, the model performs again really nice, till the text size on the page is not too big. These kind of pages might talk about a sold real estate property, about their vision, etc.

During training my model I have removed all the stop words, punctuations, hyperlink texts, form field texts with a vocabulary size of 1000. I have not done any lemmatization.

  • How can I improve the classification when the model has a high tendency to identify these index pages (the one having a lot many details about multiple real estates) as also a real estate?
  • Should I reduce the vocabulary size, as when the text extracted from the webpage is too big, it is identified as real estate?

I referred to this link to build and train my model.

$\endgroup$

1 Answer 1

0
$\begingroup$

I'd rather use something like vectorization with Sentence-BERT,to calculate a similarity score of target web page text with some reference dataset. Reference dataset should contain texts for different classes of pages.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.