NLP for appartement ads [closed]

Question

I want to analyse the Price situation for flat/appartements in my City.

How can I process the Text from flat ads (like in newspapers or online with its typical structure!) to get Features like size or number of rooms?

Is it simple to take a Package in R or Python (what is suitable?) and to Train the typical structure?

Answer 1

Look for entity recognition. The entities in your case would be price, number of bedrooms, size etcetera. What I would do is label a number of training examples with:

• From character 12 to character 22 indicates the number of bedrooms
• From character 39 to character 41 indicates the price

Etcetera.

After this you can train a model to classify entities. Sometimes the number of bedrooms will be in text instead of numbers so then you would need to write something that translates it to numerical values but you could likely just use some dictionary to do this for you.

Entity recognition models are plenty, but I think a character based LSTM approach could work well if your texts are not too long.

Answer 2

You can use Apache OpenNLP for your purpose. First, you need to train your model with training data. In your case the training data may look like:

<start:size> Big <end> Apartment at <start:address>Pavelle Road <end> on sale at just <start:price> $400k <end>.

If you can manage to have a large collection of annotated data like this, you can use the model to get data thereon. For more information look in Apache OpenNLP, its Open Source.

Answer 3

Contrary to what others are suggesting, trying to extract data on square footage and number of rooms from apartment ads is not a problem for machine learning - especially not when you don't have any training data. You will spend 10 times as long trying to build a machine learning system for this task than you would manually extracting the data yourself.

If you really want an automated system, consider building some simple regex rules based on ads you have seen. There are only so many ways people describe these features, and you should be able to get most of them when they are present in the ad. You won't get 100% accuracy, but you're not going to do any better another way, in my opinion.

You can start with some simple rules, and improve them as you look at more and more ads. In Python, for example:

import re

ad = '$3582 / 1br 1ba - 724ft2 - Breathtaking Seaport District'

sqft_pattern = '([0-9]+)\s?(sq|ft)'
beds_pattern = '(([1-9](\.5)?)\s?(br|bd|beds?|bedrooms?))|(beds|bedrooms):?\s?[1-9]'
bath_pattern = '(([1-9](\.5)?)\s?(ba|baths?|bathrooms?))|(baths|bathrooms):?\s?[1-9]'

m = re.search(sqft_pattern, ad)
sqft = m.group(1) if m else None
m = re.search(beds_pattern, ad)
beds = m.group(2) if m else None
m = re.search(bath_pattern, ad)
bath = m.group(2) if m else None

print('square  feet: {}'.format(sqft))
print('bedrooms: {}'.format(beds))
print('bathrooms: {}'.format(bath))

Output:

square feet: 724
bedrooms: 1
bathrooms: 1