11 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
noe's user avatar
  • 26.6k
5 votes
Accepted

LLMs for text generation

Yes, there are open multimodal LLMs that you can fine-tune yourself, like LlaVa, NextGPT, IDEFICS or SPHINX. Closed multimodal LLMs like GPT-4v don't offer a way to fine-tune them yet.
noe's user avatar
  • 26.6k
4 votes

From regression neural network to generative one

There's more than one type of generative network. However, I am not aware of a generic approach that can take a trained RNN-based network and essentially run it backwards to sample an input that is ...
Neil Slater's user avatar
  • 28.9k
4 votes
Accepted

Preprocessing advice for large text corpus in natural language generation (NLG)

Some comments: With Transformers and subword vocabularies (e.g. byte-pair encoding (BPE)), usually there is no need to remove named entities because the model learns to handle them just fine. For ...
noe's user avatar
  • 26.6k
3 votes

NLP - Paraphrase extraction in Python

Paraphrase detection is still a very active and very challenging research area, so it's unlikely that there are full-fledged standard libraries for this task since there is still no clear "best ...
Erwan's user avatar
  • 25.3k
3 votes
Accepted

Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

About the need for tgt_key_padding_mask While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before ...
noe's user avatar
  • 26.6k
3 votes
Accepted

Gumbel Softmax vs Vanilla Softmax for GAN training

Passing directly the output of the softmax is also common (among the few textual GANs out there), e.g. see the improved Wasserstein GANs (WGAN-GP). With hard Gumbel-softmax (+ straight-through ...
noe's user avatar
  • 26.6k
3 votes
Accepted

Which is better: GPT or RelGAN for text generation?

According to [Caccia et al., 2018], in general textual GANs are no rival for LMs regarding several quality measures. These are the conclusions of the paper: This research demonstrates that well-...
noe's user avatar
  • 26.6k
3 votes

How do we evaluate the outputs of text generation models?

Well, you missed the old good human evaluation, which is the only actual measure that can actually be trusted in terms of semantic evaluation. Also, in the reference n-gram matching area you missed ...
noe's user avatar
  • 26.6k
3 votes
Accepted

How to select the optimal beam size for beam search?

Large beam sizes do not lead to improvements but to degradation in the generated text quality, as described in the article Empirical Analysis of Beam Search Performance Degradation in Neural Sequence ...
noe's user avatar
  • 26.6k
2 votes

Choosing the size of Character Embedding for Language Generation models

There is a theoretical lower bound for embedding dimension I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics GLOVE paper discussed ...
Noah Weber's user avatar
  • 5,669
2 votes

Can BERT/ELMo be used (or retrained) to generate a text in both directions?

Neither BERT nor ELMo can be used as-is for next word (or previous word) predictions. BERT is trained on a masked language model (LM) task and can therefore only be used to guess masked words, ...
noe's user avatar
  • 26.6k
2 votes

Which NN architecture solves my problem?

Welcome to SE:DataScience. What you are looking for is called image captioning. A common approach for this is called an encoder-decoder model, where the encoder is a CNN-based NN to encode the ...
user12075's user avatar
  • 2,264
2 votes
Accepted

Text generation using Tensor Factorization

Tensor Factorization would not work for text generation as a stand-alone technique. There is no way for the decomposition to model long-term dependencies in language. Without modeling long-term ...
Brian Spiering's user avatar
2 votes
Accepted

Distinguish randomly generated texts from reasonable for human texts

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not. In order to find an ...
noe's user avatar
  • 26.6k
2 votes

Generation of medical institution names: training corpora?

I have found a way to create the corpus of potentially medical institutions by requesting the NCBI RESTful server, following the description in this link. First, you send an ESearch request containing ...
Stanislav Koncebovski's user avatar
2 votes

Generation of medical institution names: training corpora?

I can think of a couple options to collect a sample of medical institutions: Wikipedia has a list of hospitals by country (isn't Wikipedia amazing?) Many countries have some kind of national ...
Erwan's user avatar
  • 25.3k
2 votes
Accepted

Advantages of different tokenizers for NLP (specifically text generation)

The problem is of text generation. I am assuming you are trying for chatbot etc where input is a natural lanugae and output is a natural language. Since input is a natural language, all punctuations,...
amol goel's user avatar
  • 341
2 votes
Accepted

Abstracted text summarisation and generation from weighted keywords

If you have data with a good score system, I would start with something simple, because using a neural network like Bert might be complex to set up. Something simple is to take the scores and build a ...
Nicolas Martin's user avatar
2 votes
Accepted

Models that are good for long answer generation given context and question and what datasets would be the best for training?

After the comment from Nicolas Martin, I found gpt2 for qa pair generation which gave reasonable steps on how to utilize Question and Answering for GPT models and then I can specify min_length and ...
Thamognya Kodi's user avatar
1 vote
Accepted

Is there a machine learning model that is able to take reviews as input and output a new and unique blog article from them?

There is a solution that could work well. It requires minimum effort but has to be tested. If you take several reviews and you group all first paragraphs together, then the second ones, etc. and you ...
Nicolas Martin's user avatar
1 vote

Next-word Generation in Tabular Dataset

The model is currently predicting the next single character. The output of the model should be the next token.
Brian Spiering's user avatar
1 vote
Accepted

Why do RNN text generation models treat word prediction as a classification task?

The main difference between RNN-based text generation and BERT is the attention mechanism based on transformers. This attention mechanism is very important to add context between words and explains ...
Nicolas Martin's user avatar
1 vote

Distinguish randomly generated texts from reasonable for human texts

Assuming that the "human readable" texts are more likely to contain actual words, you could count the number of dictionary words that occur in each. You could use Wordnet for example. The ...
ajerneck's user avatar
  • 111
1 vote

Create an RNN on text sources with different lengths

I think that if you append a token <EOS> (end of sentence) at the end of each sentence when you merge, this would not be a problem, because the RNN would ...
Adam Oudad's user avatar
  • 1,083
1 vote
Accepted

Based on transformer, how to improve the text generation results?

If you have a lot of data available to train, you should apply the techniques used in large transformer models, like GPT-2: very deep models (48 layers for the 1.5B parameters), modified ...
noe's user avatar
  • 26.6k
1 vote

Generation of institution names

If you want to build your own dataset, you could look at packages such as: Faker Mimesis They both provide features to generate company/institution names based on certain locales as well. If your ...
Valentin Calomme's user avatar
1 vote

Generation of institution names

Conditional on you having data, yes, you can. Check out Generative Adversarial Networks and/or Reinforcement Learning for text generation. This paper is a good starting point: https://openreview.net/...
Guillermo Mosse's user avatar
1 vote

Generate text using user-supplied keywords

Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.
Pooja Sonkar's user avatar
1 vote

Is it a Problem if Training Data and Evaluation Data are Very Similar?

In general, in Machine Learning, if the train set and test set are very similar, then it leads to a small ability to generalization. In other words, the model not performing well on new data. It's ...
fuwiak's user avatar
  • 1,373

Only top scored, non community-wiki answers of a minimum length are eligible