# Tag Info

4

Its huge discrepancy, I suspect a lie. While +33% can be achieved, you said that you tried very different architectures and you did not get even close. Dont expect that one tweak, one layer, one xyz can give you all of a suddenly such a huge increase. If you did not get closer using suggestions from the paper there is also a possibility (not saying they did,...

4

There is a theoretical lower bound for embedding dimension I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics GLOVE paper discussed embedding, check page 7 for graphs. What I want to say with this reference is that you can treat it as hyperparameter and find your optimal value. EDIT: Here is my ...

3

The spaCy Python package might work for you. It allows you to easily "install" large pre-trained language models, and it provides a nice high-level interface for comparing word vectors. To install spaCy: pip install spacy Then you need to download a language model. I believe these models are trained on Common Crawl, which is a massive dataset. You ...

3

I don't think there's anything close to doing this: It would be very hard to even define the task objectively, as different humans wouldn't agree about what is credible or not. It would require a complex system to represent reliable background knowledge... and again people wouldn't agree what should be considered "reliable" or not. Generally the state of ...

3

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...

2

You lose the point of word matrix if you do that. Whole point is, very roughly speaking, when you pre-deterimened your word dictionary to calculate distributed representation of words. In other words information of one word will be in other words-vectors also.

2

You can leverage word-vector similarity in embedding models. TL;DR similiar vectors of words (for example fruits) will be clustered together in this high (vector) dimensional space. For every possible class-set you will have a class-set representative (centroid) that is actually a key (so in your case fruit, vegetable etc) all you need to do is train/find a ...

2

LDA, Latent Dirchlet Allocation, also allows to specify number of topics.

2

You sure can, for example in latent semantic analysis you can fixate number of topics (which is actually size of the decomposition matrix) beforehand.

2

IT is very similiar to PMI, here you just expand it to the whole dictionary matrix (matrix representation of the whole vocabulary), normalize it by subtracting quantitive representation of the sum of words found in row i column j and than standardize. (Like when using sklearn Standardize(), similiar atleast) Intuition? Well why is tf-idf working (generally ...

2

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT. https://arxiv.org/abs/1908.10084 They also have a github repo which is easy to work with: https://github.com/UKPLab/sentence-transformers

2

If I understand correctly, you're trying to map abstracts to their research papers. Here is a simple starting point: Compute a TF IDF model using the entire corpus (all abstracts + research papers). Use this model to transform your abstracts and research papers into a weighted vector representation. Under the TF IDF weighting scheme, these documents will ...

2

Character keyboard position information is an example of noisy channel model information, an error that depends on how a word is transmitted. It is very common to add noisy channel model information to spell checkers, including spell checkers that use character-level embeddings. Most character-level embedding models would automatically learn to model common ...

2

The answer is extensively summarized in this recent 14-pages research paper "Veracity assessment of online data". Main points: "Three main veracity assessment research directions found, i.e., utilizing implicit features, employing explicit fact checking, and the appeal to authority method." "The veracity assessment domain is still relatively immature."

2

Neither BERT nor GPT-2 are not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text.

1

Chatbots and Q&A systems differ in their complexity as well as use cases. Let's discuss each of them separately. Chatbots: They can answer various questions asked during an interactive conversation. Interactive conversion means the system keeps a track of questions asked earlier and can engage in longer conversations. They have a sought of memory which ...

1

Question-answering (QA) is sometimes used to refer to the task where the input to the system is a question and a list of possible answers (normally only a handful) or a paragraph where the answer is supposed to be found, and the expected answer is the index of the correct answer or the start/end positions where the answer located within the text. In ...

1

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.

1

It's not simple, but doable. I suggest you to create training data in the following way: take a text corpus, as large as possibile, and remove words sampled randomly. Then train an seq2seq RNN to map this "deteriorated" text with its original. The RNN you need won't be too different from an NMT model, but it's goal is different of course. It's the first ...

1

This task looks similar to what is called text segmentation, in particular topic segmentation. I don't know any python package to do it but apparently Google gives a good few results for "semantic text segmentation python" (I'm not sure that this is the best phrase, you might want to try variations). Note: this is still an active NLP research topic as far ...

1

A direct way to find the words which are the most representative of a class is to calculate the probability of the class given a word: $$p(c|w)=\frac{\#\{\ d\ |\ label(d)=c\ \land w\in d\}}{\#\{\ d\ |\ w\in d\ \}}$$ Ranking the words according to their probability $p(c|w)$ gives: highest values: the most correlated words for the class lowest values: the ...

1

I think here you must maintain the actual tf-idf and create corpus over it.. Assuming you already have lables for documents available. You can rum classification over it. Best classification I am anticipating for this problem would be naive bayes..

1

Sadly, I don't think that Multilingual BERT is the magic bullet that you hoped for. As you can see in Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (Wu and Dredze, 2019), the mBERT was not trained with any explicit cross-lingual task (for example, predicting a sentence from one language given a sentence from another language). ...

1

I am not sure you need to build an algorithm to 'learn' that. Unless you want to do very advanced work on fields of science classification you can get existing trees. Some that come to mind : https://arxiv.org/ has fields classified on their home page. I even remember they did some network analysis to observe the relevence of their classification. Doing ...

1

You mentioned: My understanding is that standard NLP techniques may not fit here, because the relationship between sub-fields and fields are linked by its meanings but not word-frequency or word-embedding etc. However, your understanding is not totally correct, because word embeddings do convey meaning in them and could be used in your case. Here is an ...

1

You need an ontology or any form of third-party data which describes the relationships between fields and sub-fields. You could use resources such as Wikipedia categories or standard library classifications for instance. There are probably other options for scientific fields.

1

When using a RNN, you don't feed all the data at once, you usually have a seq2seq model. The models are created with an encoder-decoder architecture. The LSTM is used in the encoding phase. So, let's say you have a text of 78 words. You will feed the embedding vector (size 300) of those 78 words, 1-by-1 into your LSTM and in the end you will get a hidden ...

1

@Ravikm, excellent question. In Spacy, you can assign a word manually. For example, "Tesla" to ORG. Source: screenshot from Jose Portilla's NLP course on Udemy.

1

I would start by training some very strong Named Entity classifier on available datasets for NER. One is the Annotated Corpus for Named Entity Recognition available on Kaggle. Additionally, you can find a good list of datasets here. I know they have nothing to do with cybersecurity, but I think it's important to incorporate very different sources in a big, ...

Only top voted, non community-wiki answers of a minimum length are eligible