People who code: we want your input. Take the Survey
4

There's more than one type of generative network. However, I am not aware of a generic approach that can take a trained RNN-based network and essentially run it backwards to sample an input that is expected to produce a given output. So I am suggesting a couple of generative approaches that I have seen working, but that will require that you construct and ...


3

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-based decoders are autoregressive by nature, Transformers are not. Instead, all predictions are generated at once based on the real target tokens (i.e. teacher ...


3

Passing directly the output of the softmax is also common (among the few textual GANs out there), e.g. see the improved Wasserstein GANs (WGAN-GP). With hard Gumbel-softmax (+ straight-through estimator), you pass one-hot encoded vectors, which is the same as what you have with real data. If you pass the output of the softmax, the discriminator should be ...


2

According to [Caccia et al., 2018], in general textual GANs are no rival for LMs regarding several quality measures. These are the conclusions of the paper: This research demonstrates that well-adjusted language models are a remarkably strong baseline and that temperature sweeping can provide a very clear characterization of model performance. A well-...


2

Welcome to SE:DataScience. What you are looking for is called image captioning. A common approach for this is called an encoder-decoder model, where the encoder is a CNN-based NN to encode the content of an image into a vector representation, and a RNN-based decoder generates its caption conditioned on that vector. This is a relatively well-studied area, ...


2

Tensor Factorization would not work for text generation as a stand-alone technique. There is no way for the decomposition to model long-term dependencies in language. Without modeling long-term language dependencies, its results would be similar to low-order Markov chains. Tensor Factorization could be used as another signal in a larger natural language ...


2

Paraphrase detection is still a very active and very challenging research area, so it's unlikely that there are full-fledged standard libraries for this task since there is still no clear "best solution" to this problem. In order to build a corpus you might want to look at how shared tasks/competitions have done it before. I know at least of SemEval which ...


2

There is a theoretical lower bound for embedding dimension I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics GLOVE paper discussed embedding, check page 7 for graphs. What I want to say with this reference is that you can treat it as hyperparameter and find your optimal value. EDIT: Here is my ...


2

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not. In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset. UPDATE: There are multiple ...


2

I have found a way to create the corpus of potentially medical institutions by requesting the NCBI RESTful server, following the description in this link. First, you send an ESearch request containing some searching criteria (e.g. 'radiology', 'dicom', 'segmentation' - or whatever). As a response you obtain an XML document with a list of PubMed Ids. Then you ...


2

I can think of a couple options to collect a sample of medical institutions: Wikipedia has a list of hospitals by country (isn't Wikipedia amazing?) Many countries have some kind of national directory of medical institutions, but that would probably be difficult to scrap and specific to each country. UMLS has a category ("semantic group") for &...


2

About the need for tgt_key_padding_mask While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this. For left padding to be handled correctly, you must mask the padding tokens, because the ...


1

Assuming that the "human readable" texts are more likely to contain actual words, you could count the number of dictionary words that occur in each. You could use Wordnet for example. The number or proportion of word hits, and their length, could be features for a model or maybe it would be enough with a simple cutoff rule. You might want to ...


1

If you have a lot of data available to train, you should apply the techniques used in large transformer models, like GPT-2: very deep models (48 layers for the 1.5B parameters), modified initialization, pre-normalization, and reversible tokenization. You could also apply GPT-3's locally banded sparse attention patterns. If you have very small training data, ...


1

If you want to build your own dataset, you could look at packages such as: Faker Mimesis They both provide features to generate company/institution names based on certain locales as well. If your goal is to generate training data for a NER task, this should be a good start. If it's to generate company names, this will already cover quite a bit.


1

Conditional on you having data, yes, you can. Check out Generative Adversarial Networks and/or Reinforcement Learning for text generation. This paper is a good starting point: https://openreview.net/forum?id=rJedV3R5tm. Also, here's a tool that might help you. What you can do is generat these institution names without differentiating by type, and then build ...


1

Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.


1

In general, in Machine Learning, if the train set and test set are very similar, then it leads to a small ability to generalization. In other words, the model not performing well on new data. It's called overfitting. In your case, could be happened more, if some rows are the same in train and data set, then we could say that it's data leakage.


1

Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-training can be reused to produce meaningful sentences.


1

Neither BERT nor ELMo can be used as-is for next word (or previous word) predictions. BERT is trained on a masked language model (LM) task and can therefore only be used to guess masked words, obtaining this way a contextual representation of them. There have been some attempts to use BERT for text generation, but they have been unsuccessful up to now. ELMo ...


1

A starting point that comes to mind is creating a cost function for a sentence being in IP. Now, while normally this is a binary affair (either a sentence is in IP or not - or so I would assume), this does not lend itself readily to the task. You should devise a cost function that measures how close your sentence is to IP (so I assume that sentence with 11 ...


1

First off, I think that since the goal of your model will be to generate new names based on a description, your model should work at a character-level and not word-level. You can think of the level at which your model is working as the building blocks you are providing for it (it needs to learn them during training). These building blocks are than used for ...


1

From the link you provided: Sample a sequence of characters according to a sequence of probability distributions output of the RNN Arguments: parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b. char_to_ix -- python dictionary mapping each character to an index. Returns: indices -- a list of ...


1

You can use word embedding, to encode words as vectors of real numbers. Then all calculations, such as comparison of words (to find similarity), are performed in that high-dimensional space. "What would be the best way to format my input (and my output) for this problem?" I cannot tell which is the best approach (depends on your problem) but this one ...


1

I didn't find any good explanations or papers on this topic, other than things about category based models and multigrams (parts of words). So, I came up with one myself. I'm using Java, but here's the code for the length normalization translated to Python. Also, this is assuming that you're adding the log probabilities for each word together, like KenLM ...


1

I think you might want to check Andrej Karpathy's work around charRNNs some pretty cool work is being done. The github link has all the relevant code too. If you are looking for a more applied way - you can check my blog on deep learning gender from name which effectively uses character level LSTM RNNs to learn patterns : medium.com/@prdeepak.babu/deep-...


Only top voted, non community-wiki answers of a minimum length are eligible