8
votes
Accepted
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
- 20.7k
4
votes
From regression neural network to generative one
There's more than one type of generative network. However, I am not aware of a generic approach that can take a trained RNN-based network and essentially run it backwards to sample an input that is ...
- 28.3k
4
votes
Accepted
Preprocessing advice for large text corpus in natural language generation (NLG)
Some comments:
With Transformers and subword vocabularies (e.g. byte-pair encoding (BPE)), usually there is no need to remove named entities because the model learns to handle them just fine. For ...
- 20.7k
3
votes
Accepted
Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder
About the need for tgt_key_padding_mask
While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before ...
- 20.7k
3
votes
Accepted
Gumbel Softmax vs Vanilla Softmax for GAN training
Passing directly the output of the softmax is also common (among the few textual GANs out there), e.g. see the improved Wasserstein GANs (WGAN-GP).
With hard Gumbel-softmax (+ straight-through ...
- 20.7k
3
votes
Accepted
Which is better: GPT or RelGAN for text generation?
According to [Caccia et al., 2018], in general textual GANs are no rival for LMs regarding several quality measures. These are the conclusions of the paper:
This research demonstrates that well-...
- 20.7k
3
votes
NLP - Paraphrase extraction in Python
Paraphrase detection is still a very active and very challenging research area, so it's unlikely that there are full-fledged standard libraries for this task since there is still no clear "best ...
- 24.6k
3
votes
How do we evaluate the outputs of text generation models?
Well, you missed the old good human evaluation, which is the only actual measure that can actually be trusted in terms of semantic evaluation. Also, in the reference n-gram matching area you missed ...
- 20.7k
2
votes
Choosing the size of Character Embedding for Language Generation models
There is a theoretical lower bound for embedding dimension
I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics
GLOVE paper discussed ...
- 5,539
2
votes
Can BERT/ELMo be used (or retrained) to generate a text in both directions?
Neither BERT nor ELMo can be used as-is for next word (or previous word) predictions. BERT is trained on a masked language model (LM) task and can therefore only be used to guess masked words, ...
- 20.7k
2
votes
Which NN architecture solves my problem?
Welcome to SE:DataScience.
What you are looking for is called image captioning. A common approach for this is called an encoder-decoder model, where the encoder is a CNN-based NN to encode the ...
- 2,159
2
votes
Accepted
Text generation using Tensor Factorization
Tensor Factorization would not work for text generation as a stand-alone technique. There is no way for the decomposition to model long-term dependencies in language. Without modeling long-term ...
- 19.8k
2
votes
Accepted
Distinguish randomly generated texts from reasonable for human texts
You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not.
In order to find an ...
- 20.7k
2
votes
Generation of medical institution names: training corpora?
I have found a way to create the corpus of potentially medical institutions by requesting the NCBI RESTful server, following the description in this link.
First, you send an ESearch request containing ...
2
votes
Generation of medical institution names: training corpora?
I can think of a couple options to collect a sample of medical institutions:
Wikipedia has a list of hospitals by country (isn't Wikipedia amazing?)
Many countries have some kind of national ...
- 24.6k
2
votes
Accepted
Abstracted text summarisation and generation from weighted keywords
If you have data with a good score system, I would start with something simple, because using a neural network like Bert might be complex to set up.
Something simple is to take the scores and build a ...
- 4,454
2
votes
Accepted
Advantages of different tokenizers for NLP (specifically text generation)
The problem is of text generation. I am assuming you are trying for chatbot etc where input is a natural lanugae and output is a natural language.
Since input is a natural language, all punctuations,...
- 321
2
votes
Accepted
Models that are good for long answer generation given context and question and what datasets would be the best for training?
After the comment from Nicolas Martin, I found gpt2 for qa pair generation which gave reasonable steps on how to utilize Question and Answering for GPT models and then I can specify min_length and ...
1
vote
Accepted
Is there a machine learning model that is able to take reviews as input and output a new and unique blog article from them?
There is a solution that could work well. It requires minimum effort but has to be tested.
If you take several reviews and you group all first paragraphs together, then the second ones, etc. and you ...
- 4,454
1
vote
Next-word Generation in Tabular Dataset
The model is currently predicting the next single character. The output of the model should be the next token.
- 19.8k
1
vote
Accepted
Why do RNN text generation models treat word prediction as a classification task?
The main difference between RNN-based text generation and BERT is the attention mechanism based on transformers.
This attention mechanism is very important to add context between words and explains ...
- 4,454
1
vote
Distinguish randomly generated texts from reasonable for human texts
Assuming that the "human readable" texts are more likely to contain actual words, you could count the number of dictionary words that occur in each.
You could use Wordnet for example.
The ...
- 111
1
vote
Accepted
Based on transformer, how to improve the text generation results?
If you have a lot of data available to train, you should apply the techniques used in large transformer models, like GPT-2: very deep models (48 layers for the 1.5B parameters), modified ...
- 20.7k
1
vote
Generation of institution names
If you want to build your own dataset, you could look at packages such as:
Faker
Mimesis
They both provide features to generate company/institution names based on certain locales as well.
If your ...
- 5,346
1
vote
Generation of institution names
Conditional on you having data, yes, you can. Check out Generative Adversarial Networks and/or Reinforcement Learning for text generation. This paper is a good starting point: https://openreview.net/...
- 335
1
vote
Generate text using user-supplied keywords
Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.
- 11
1
vote
Is it a Problem if Training Data and Evaluation Data are Very Similar?
In general, in Machine Learning, if the train set and test set are very similar, then it leads to a small ability to generalization. In other words, the model not performing well on new data. It's ...
- 1,355
1
vote
How to generate a sentence with exactly N words?
Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-...
- 5,539
1
vote
NLP text autoencoder that generates text in poetic meter
A starting point that comes to mind is creating a cost function for a sentence being in IP. Now, while normally this is a binary affair (either a sentence is in IP or not - or so I would assume), this ...
- 604
1
vote
What NN architecture to predict fantasy character names based on description?
First off, I think that since the goal of your model will be to generate new names based on a description, your model should work at a character-level and not word-level.
You can think of the level ...
- 2,180
Only top scored, non community-wiki answers of a minimum length are eligible