3
$\begingroup$

I was looking at Google's Smart linkify machine learning models, as it closely relates to a personal project. And couldn't quite understand how the features are fed to the neural network.

It's about the following:

enter image description here

Given a candidate entity span, we extract: Left context: five words before the entity, Entity start: first three words of the entity, Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many), Right context: five words after the entity, Entity content: bag of words inside the entity and Entity length: size of the entity in number of words. They are then concatenated together and fed as an input to the neural network.

I have 4 primary questions which I cant find a clear answer to:

  • The article specifies the features are concatenated. How does a concatenation layer work internally? Does it concatenate all the values in a single variable, in a very literal sense? how does that work computationally?

  • How can a Bag of Words be a feature, when its a key-value pair? Or is it also just all concatenated into one variable. Which again, how can that work computationally?

  • The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?

  • Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features?

Perhaps a simple keras model with some hardcoded input variables would help shape the answer.

Is there more material online to understand and recreate the entity recognition model?

EDIT:

The article also mentions the lack of available context or entities. E.g. the diagram shows 4 features for entity (2 for entity start, 2 for entity end). The article mentions duplication, but duplicating 3 times doesnt sound like a great idea. Would a convolution layer with a filter (1x3) work better?

And how would a Keras model look like then? Would it have two separate input layers? One input layer with 10 input features for the context + 1 feature for BoW + 1 feature for entity length. And another input layer with 4 input features followed by a convolution layer. And then both layers lead to a concatenate layer?

Improve this question
$\endgroup$
1
  • $\begingroup$ You could ask the poster, Lukas Zilka. He is present on LinkedIn, has his own website where his email is stated. $\endgroup$
    – keiv.fly
    Feb 28, 2020 at 23:25

2 Answers 2

Reset to default
2
+50
$\begingroup$

Let me take a crack at your questions:

The article specifies the features are concatenated. How does a concatenation layer work internally? Does it concatenate all the values in a single variable, in a very literal sense? how does that work computationally?

The concantenation of information in this context is, concatenation of vector represententations of the text. You could concatenate using a concatenation layer as described here. This is a very common approach followed where you want to feed your network information by taking various contexts depending on the problem you need to solve.

How can a Bag of Words be a feature, when its a key-value pair? Or is it also just all concatenated into one variable. Which again, how can that work computationally?

Bag of words, is typically a vector representation of the context. The above answer should help you. You could take a look at how to combine embeddings here as well

The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?

Yes, you are spot on. You combine the embedding vectors that you generate using skipgram or cbow, or you could even use one-hot encoded vectors.

Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features?

Its always good to pad and use well defined dimensional vectors. Helps you structure your architecture better. You could use masking layer to ensure that the network ignores the paddings.

I hope this helps.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you. I have a few more clarification questions. - The vector representations of the text could, for example, be a simple word2vec layer? - The BoW is only based on the entity right, so basically it only serves as a count to help the neural network when an entity has to de duplicated to fill a missing spot? - you mention combining the vectors using skipgram or cbow, this is handled by the concatenation layer (keras etc) right? I assume a configuration I can tweak. Or is it something I have to do as preprocessing? My last question is edited in the post, it was too long. Thank you $\endgroup$
    – Rien
    Feb 29, 2020 at 11:03
  • $\begingroup$ The vector representation could be generated and fed as input to your network via using word2vec or fasttext or one hot encoding using sklearn. Its just a vector, you can create it however you want to. What do you mean the BoW is only based on the entity right? Every word/entity is represented as a vector based on its neighbors or with one-hot and a label. $\endgroup$
    – Nischal Hp
    Mar 1, 2020 at 20:36
  • $\begingroup$ With regards to missing spots/ lesser words, this is exactly the reason you pad, so that the dimensions of vectors you feed into your network always have the same size. You concatenate vectors after padding or before padding and then pad the output of concatenation. Ideally, I would suggest padding before concatenation. $\endgroup$
    – Nischal Hp
    Mar 1, 2020 at 20:37
  • $\begingroup$ Only padding im familiar with is in convolutional layers. Is this the padding type refered to? Such as 0 padding, just creating vectors with the same dimensions but filled with 0's? $\endgroup$
    – Rien
    Mar 1, 2020 at 20:42
  • $\begingroup$ Yes, ideally, when you create vectors, you specify a dimension, so the necessity of padding vectors reduces. Padding is useful, when you need to fill the size of statements. For example: Let us say, you have sentences with 30 words and sentences with 100 words. If you end up deciding to use only 70 words, then oyu need to pad when you come across sentences that have < 70 words. $\endgroup$
    – Nischal Hp
    Mar 1, 2020 at 20:49
0
$\begingroup$

Bag of words representation

Tackling your questions one-by-one, a classic way to represent bag of words in DNN solutions is not as a key-value representation, but as a "n-hot" vector. So if the entity consists of tokens #123, #456 and #567 (from a pre-computed dictionary), then you can have a vector with the length of the dictionary where all values are 0 but the bits #123, #456 and #567 are set to one.

If precomputed embeddings are used, another option that I have seen is to add or average the embeddings for these multiple tokens - that way you have a fixed length representation of the entity, no matter if it has 1 token or 5 tokens.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Could you elaborate on the last part? Inconsistent number of / length of the entity is one of my concerns $\endgroup$
    – Rien
    Feb 29, 2020 at 15:09
  • $\begingroup$ There's not much to elaborate - "add or average the embeddings" means literally just that, if you've got three vectors each with 100 elements, then you can do elementwise addition (or average) and you'll get one vector of 100 elements that will be still mapped to the same 100-dimensional embedding semantic space. $\endgroup$
    – Peteris
    Feb 29, 2020 at 15:17
  • $\begingroup$ Does that work properly for numerical entities too? Embedding numerical entities and then averaging em? $\endgroup$
    – Rien
    Feb 29, 2020 at 15:22
  • $\begingroup$ As far as NLP representations (no matter if dictionaries or word embeddings) are concerned, there's absolutely no technical difference between numerical entities or letter-based entities or words or subword units/bytepair encoding or chinese characters or bytes from a string or DNA sequence letters or something else. A token is a token. Some styles of tokenization work better than others for different tasks, but after you've got a sequence of tokens, then the rest of the model would treat it as a sequence of tokens. $\endgroup$
    – Peteris
    Feb 29, 2020 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.