# Making a tagged Part-of-Speech corpus with the help of a lexicon

I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe.

Also, I have a list of tokens (words) in another Dataframe.

I want to take each token in the untagged corpus and search it inside the entire lexicon. If a token is matched in the lexicon, then take that token's tag and add it to another column in the untagged Dataframe. If the token is not found then return 'X'.

Here how I did it:

lexicon_rows = lexicon.iloc[:,:].values

tag = lexicon_rows[:, [1]][lexicon_rows[:, 0] == untagged_row['word']]
if tag.size == 1:
return str(tag[0][0])
else:
return 'X'



I am not sure whether each word in the untagged corpus is searched against the entire lexicon or not.

My question is: Am I doing it right? If so, what is a better approach to accomplish this task? If not, could you please provide me with an answer?

Thank you.

• The logic looks correct to me, but I'm not familiar with pandas. But isn't it possible that the same word appears in different rows with different POS tags in the lexicon? if it is you should take care of this case, currently your method doesn't. Also you're aware that this task is usually done with a POS tagger, right? – Erwan Nov 29 '20 at 16:51
• Thank you so much. Yes, I am aware. But first I am trying to build an annotated corpus to train the tagger. I cannot train the model with a lexicon/dictionary. – Aziz Qadeer Nov 29 '20 at 21:00
• Ok then. In case you're interested, there are POS-annotated corpora for Arabic: universaldependencies.org – Erwan Nov 29 '20 at 23:37