0
$\begingroup$

Let's say I have a db driven app which has two tables: people and organizations. I run my documents through a named entity recognition program. Now - what do I do with this additional ner information?

Does it go in my existing db tables? Do I have to rewrite my existing tables to accommodate the ner data? Do I make new tables for the ner data? Do I connect them to my older tables with foreign keys? Do I need new functions that pull response data from these new ner objects instead of, or with, my original ones? Are these new entities "objects" in the traditional OOP sense? But they are different objects from the ones in my original db tables, so do they replace the old, non-ner objects? Does it matter what kind of datastore the new ner objects go into in order to be fully utilized? rmdbs? key/value? document based?

The only thing I’ve been able to find online about this is a Medium post that pickles the model - it says nothing about any other kind of datastore - and the Flask website just displays the information by accessing it through an api.

A search here on datascience.stackexchange.com got this:

We couldn't find anything for integrate with existing website

And now I get a warning that my question "appears to be subjective and is likely to be closed". I'm sure that's machine learning at work. Granted, the step by step details might be different if your site is php vs python, but the issue is a major one I assume a lot of people are going to run into sooner if not later. My site is in Django.

How do you integrate the product of nlp into an existing, complex website with many apps and functions?

$\endgroup$
  • $\begingroup$ What do you want to do with the newly generated data? Is it a part of your functionality? $\endgroup$ – S van Balen Aug 7 '19 at 14:42
  • $\begingroup$ @SvanBalen: Yes, absolutely. The assumption is that this new data will enhance my user experience and give them deeper, richer information to use. I just don't know how I'm supposed to integrate the old with the new. If you or anyone else can point me to a resource where this integration is discussed, that would be great. $\endgroup$ – Malik A. Rumi Aug 7 '19 at 14:59
1
$\begingroup$

The answer is it depends on you architecture.

Some things to consider:

  • The tags are attributes of words in a (presumably) text field in a database. So Boyce Codd would prescribe putting it in a seperate table referencing the primary key (and possibly the position in the text)
  • The tags are just a function over your data, so storing it at all should be for caching purposes. Which you might not want to do in your database, but at the presentation level, or perhaps in a separate database, or schema.
  • The tags are in fact the product of an approximation of a function over your data. There is no such thing as a perfect NER-tagger. This means that you might want to switch the function later buy something more suited.

It could make sense to extract the information on display time from a microservice (perhaps hosted by you), and cache the resulting presentation for better performance.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ By 'tags' I assume you include NER labels like "'Apple' = ORG" in spaCy. But calling it a function really changes my perspective. I was thinking of this as the return value, not the function itself. But that's why I asked the question, I didn't understand how all the pieces fit together. I need to chew on this for a while. $\endgroup$ – Malik A. Rumi Aug 7 '19 at 16:42
  • $\begingroup$ Oops: I said Part of Speech, but we were talking about Named Entity Recognition. I have edited the answer. But yes named entities can be thought of as (xml type) tags. But again it depends on what you want to do with it. If you want to retrieve records by tag, you might still want to go with the in database approach. You might want to apply it as an index in that case. $\endgroup$ – S van Balen Aug 7 '19 at 18:58
  • $\begingroup$ Thanks. You have helped clarify a lot of things for me. $\endgroup$ – Malik A. Rumi Aug 7 '19 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.