Hot answers tagged

2

First, let's clarify the issue with one-hot vectors: most NLP neural models nowadays don't use one-hot encodings for the model input; instead, they use (non contextual) embedding layers. While theoretically you get the same result multiplying a one-hot vector with a matrix, it is more practical just to index the position in the table directly, which is what ...


2

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that. However, you should take into account that it is possible that your model generates very similar sentences ...


2

Subword tokenization is the norm nowadays in NLP models because: It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. This is a problem for morphologically-rich languages, proper nouns, etc. Subword vocabularies allow representing these words. By having subword tokens (and ...


2

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are: UDPipe – developed at Charles University in Prague. Gets very good results (at least for parsing), but has a little unintuitive API. Stanza – developed at Stanford University. The API is quite similar to Spacy. However, they ...


1

No, assuming your input vectors are one-hot encodings. These input one-hot encodings are in an $n$-dimensional Euclidean vector space. The last hidden layer of an LSTM is not due to the non-linear activation functions across the encoder. Therefore, an average of the inputs will not necessarily align well in a vector space with the model output, nor are you ...


1

There are a couple of options: Optimize tensorflow for your specific CPU. Sometimes the official versions of tensorflow are not compiled with support for some instruction sets (e.g. SSE4.1, SSE4.2, AVX, AVX2, FMA). Usually, there is a tensorflow runtime warning message stating so. This prevents some computations to take place in parallel. You can either ...


1

Which are the pros and cons of this technique? Context insensitivity: the libraries you mention are intended for general sentiment analysis so you could encounter some false positive/false negative issues. False positives: words with a particular sentiment in the dictionary that doesn't apply to headlines –e.g. "low" may have a negative ...


1

What you're proposing is some very simple form of ensemble learning. You need to have at least a sample of labelled data in order to evaluate any method. Using this labelled data you can: evaluate each of the three methods on their own evaluate your idea of averaging the 3 methods predictions if you have enough labelled data, you could even train a model ...


1

They might or might not be similar, the embeddings extracted by mean pooling the BERT output usually have high cosine similarity even though the input sentences are completely different. Bert embeddings are not meant for sentence similarity task(SST), but there is some research combining Bert and SST. Here are those resources, SBERT paper: https://arxiv.org/...


1

I agree with your assumption, the vector space is the same so I don't see any major problem with this approach. Still this approach might cause some more subtle bias, depending on the differences between the models (sets of terms, number of clusters). I could imagine the following problems happening: if there is a big difference in number of clusters ...


1

I think there are (at least) two parts to take into account in evaluating such a model: Whether the generated text correctly relate to the input topic Whether the generated text is grammatically and semantically acceptable In my opinion the first kind of evaluation could reasonably be done with an automatic method such as the one you propose. Note that ...


1

In general this is a difficult problem, it's about the problem of Natural Language Understanding which is far from being solved. The advanced option requires a full syntactic parsing of the sentence, ideally followed by some kind of semantic representation of the sentence, for example by extracting relations. As far as I know this is rarely used because ...


1

Is this a fair assumption? No: Named Entity Recognition (NER) is a specific task which consists in detecting named entities. The more general term for this kind of task in Machine Learning is sequence labelling, because it's not only about classifying words but annotating a sequence of instances in which order matters (e.g. words). It's true that NER is ...


1

The main reason would be the density and diversity of sentiments in long texts. Assuming the presence of a certain sentiment (positive, negative), it can be measured easier within a short text as the probability of having more than one subject or more than one specific sentiment about a subject, is less. If you read a long text, there might be several ...


1

I was wondering if it is just because of the computational power and time required to train ML algorithms It is not because of that; it is arguably because a long and structured text may probably contain segments of "positive" sentiment along with "negative" ones, it can be infinitely more subtle and nuanced, and in principle trying to ...


1

One option is the ploygot package which can perform morphological analysis in English and Hindi. from polyglot.text import Word word = Word("Independently", language="en") print(word, w.morphemes)


1

At training time, the input to the decoder is the target sentence tokens, which are indeed unknown at the test time. What you call the second input are the desired outputs, which are not usually referred to as an input to the decoder, 1. for clarity, 2. they are technically input to the loss function. At test time, we do not need the loss function, but we ...


1

Speaking about vanilla BERT. It is currently not possible to fine-tune BERT-Large using a GPU with 12GB - 16GB of RAM, because the maximum batch size that can fit in memory is too small (even with batch size = 1). The fine-tuning examples which use BERT-Base should be able to run on a GPU that has at least 12GB of RAM using the hyperparameters given on this ...


1

I think the function get_close_matches in module difflib could be more suitable for such a requirement. get_close_matches(word, possibilities, n=3, cutoff=0.7) possibilities -> is the list of words n = maximum number of close matches cutoff = accuracy of matches. data=["drain","rain","brain","stackexchange"] word=...


Only top voted, non community-wiki answers of a minimum length are eligible