I have a dataset with two columns. First column has some text (news article) and the second column contains names of people (not exactly English names) present in those news articles (first column). I've tried to build a custom named entity recognizer using Spacy but it isn't working. Can I use deep learning approach to identify the names in unseen news articles ? (Test data)
Named Entity Recognition (NER) is about identifying the position of the NEs in a text. This means that each instance must represent a particular position in a text, and the NER will predict whether this position corresponds to a NE or not. Currently your data is not formatted in this way so it's not surprising that it doesn't work: in a vector representation of the whole text the model cannot find the kind of indication it needs to identify NEs.
I suggest you to take a look at the performance of mainstream taggers first. Namely:
stanfordnlp. Check how good they are on finding the right set of names on your Test set.
Alternatively, you must make your own tagger. Any NER tagger is fundamentally a classifier with RNN or CNN layers that process input text. The output, Dense layer has as many nodes as classes avilable.
Another option is to download some pretrained model. For example, Huggingface's
transformers library lets you download lots of Transformer-based architectures (like BERT), that you can then fine tune for your specific task.
However, building your own model is a lot of work. If you find that mainstream classifiers do a pretty good job, and you don't have too much time, then go with those tools. Good luck!