I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe.


Also, I have a list of tokens (words) in another Dataframe.

untagged corpus

I want to take each token in the untagged corpus and search it inside the entire lexicon. If a token is matched in the lexicon, then take that token's tag and add it to another column in the untagged Dataframe. If the token is not found then return 'X'.

Here how I did it:

lexicon_rows = lexicon.iloc[:,:].values

def add_tag(untagged_row):
    tag = lexicon_rows[:, [1]][lexicon_rows[:, 0] == untagged_row['word']]
    if tag.size == 1:
        return str(tag[0][0])
        return 'X'

untagged['tag'] = untagged.apply(add_tag, axis=1)

tagged corpus

I am not sure whether each word in the untagged corpus is searched against the entire lexicon or not.

My question is: Am I doing it right? If so, what is a better approach to accomplish this task? If not, could you please provide me with an answer?

Thank you.

  • $\begingroup$ The logic looks correct to me, but I'm not familiar with pandas. But isn't it possible that the same word appears in different rows with different POS tags in the lexicon? if it is you should take care of this case, currently your method doesn't. Also you're aware that this task is usually done with a POS tagger, right? $\endgroup$ – Erwan Nov 29 '20 at 16:51
  • $\begingroup$ Thank you so much. Yes, I am aware. But first I am trying to build an annotated corpus to train the tagger. I cannot train the model with a lexicon/dictionary. $\endgroup$ – Aziz Qadeer Nov 29 '20 at 21:00
  • $\begingroup$ Ok then. In case you're interested, there are POS-annotated corpora for Arabic: universaldependencies.org $\endgroup$ – Erwan Nov 29 '20 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.