1
$\begingroup$

I've tried stemming and lemmatization on this but nothing has quite worked so far.

How can I resolve country name and nationality as a singular entity?

For example:

Canada and Canadian should just be one entity: Canada Uganda and Ugandan should just be Uganda

It does seem like stemming is one approach here. I just have found it misses a fair number of countries.

$\endgroup$
1
$\begingroup$

I believe that Lemmatisation is the right thing. Anyways another way to do it is to use WordNet. For a word which its POS is Noun you can query if it has a member holonym, this feature will show the country which it belongs. For instance, Canadian -> Canada. Then you have to be careful, cause if you take the member holonym of Canada that will be the British Commonwealth. I guess you could have a Levenshtein distance threshold to ignore these.

You can have a look on the online web app.

$\endgroup$
2
  • $\begingroup$ Lemmatisation doesn't work, though. The Lemma of Canadian is Canadian. I will check out the holonym idea that is neat. $\endgroup$ – Learning stats by example Oct 14 '19 at 21:45
  • 1
    $\begingroup$ Oh I see, thnx and good luck. $\endgroup$ – 20roso Oct 14 '19 at 21:51
0
$\begingroup$

I see the tag is already present but you should look into Named Entity Recognition (NER). It is used to tag or find proper nouns in speech and can handle people and locations (of which countries could be thought of as). This page says the popular spaCy library specifically handles countries:

https://spacy.io/api/annotation#named-entities

It would easily catch Uganda, though you would have to experiment with the adjectives ("Ugandan").

Edit based on your comment that the resolving is the real problem. For that I would try:

  1. Use something like word2vec and cosine similarity to analyze all the "caught" words. If you load a pre-trained word2vec model and pass it the words, you'll get back a vector that represents that word. If you then compare all the vectors, I suspect Uganda and Ugandan would have very high similarity. See this answer for someone using word2vec on country names: https://stackoverflow.com/questions/21979970/how-to-use-word2vec-to-calculate-the-similarity-distance-by-giving-2-words
  2. Use a string comparison function to identify caught "locations" that are highly similar to other caught locations. This will work well for some countries (Uganda) but not others (France/French, Switzerland/Swiss).
$\endgroup$
2
  • $\begingroup$ This is precisely my problem.. I have them both caught. But not resolved.. $\endgroup$ – Learning stats by example Oct 14 '19 at 19:55
  • 1
    $\begingroup$ Ah. I will update my answer with some ideas for that. $\endgroup$ – CalZ Oct 14 '19 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.