2
$\begingroup$

I would like to extract location or contents related to location from raw text. I used the NLTK and spaCy packages already; none worked for me. For example, both would neglect 'canada' as a location because it is written in lower case format. Or, if I just include somewhere new in a text, both would fail to recognize it as a location.

Could anyone here recommend a solution (paper, blog, GitHub or anything) to solve this problem? To be more specific, I would like my algorithm to recognize "sakfhajl" and "alksjf" as locations in the example below:

"I am currently at sakfhajl street" or "I wanted to spend more time in alksjf but ..."

Yes, I know it is hard, but don't us humans do the same? We all recognize some names as locations given the context, although we might have never heard the name before.

$\endgroup$
1
$\begingroup$

You can first detect the "out of vocabulary" words, and check if they are part of a location dataset.

There are locations datasets that you can use and adapt them to be not case sensitive.

Here are the ones for the cities: https://simplemaps.com/data/world-cities

About streets, you can use the world roads dataset and apply the same logic: https://sedac.ciesin.columbia.edu/data/set/groads-global-roads-open-access-v1

$\endgroup$
4
  • $\begingroup$ To check if they are part of a location data-set, either I must do brute force or some search algorithm, so you believe the solution cannot be out of this? By the way thanks for the data-sets $\endgroup$ Jun 30 at 19:32
  • 1
    $\begingroup$ I've proposed those solutions because you seem to use NLTK and Spacy. Otherwise, if your objective is to only detect a location word in a phrase, there are deep learning classification solutions with word embedding that would be better. For example Temporal Convolutional Network (TCN) or Long Term Short Term Memory (LSTM) using small sets of words. The drawback is that they need to be trained. $\endgroup$ Jul 1 at 7:53
  • $\begingroup$ Could you drop a link so I can follow ? $\endgroup$ Jul 2 at 3:26
  • 1
0
$\begingroup$

I just found myself the perfect solution. All you need to do is to write an embedding layer and a bi-Directional LSTM mine is this,

input_layer = tf.keras.Input(shape = (max_len,))
embeding_layer = tf.keras.layers.Embedding(top_wordings, embeding_length, input_length= max_len) (input_layer)
lstm_layer = tf.keras.layers.Bidirectional( tf.keras.layers.LSTM(15, return_sequences=True) ) (embeding_layer)
out_layer = tf.keras.layers.TimeDistributed( tf.keras.layers.Dense(2, activation= "softmax") ) (lstm_layer) 
model = tf.keras.Model(input_layer, out_layer)

model.compile (optimizer= "adam", loss = "sparse_categorical_crossentropy", metrics=["accuracy"])

Then start creating random sentences yourself with location names with all location prepositions, that is sentences like this

"I am at/above/under/between somelocation"

and then change somelocation with all your location dataset names.

for the target part you only need two tag, wether it is a location or not so (0,1) then let the network train. Then it will even find locations even out of your vocabulary based on the preposition.

Hope it helps you too/ remember to make all train dataset in lower case to avoid the sensitivity

Also, please use open-street-map for extracting street names

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.