All Questions
Tagged with language-model neural-network
13 questions
0
votes
1
answer
201
views
What does Codex take as tokens?
The typical default for neural networks in natural language processing has been to take words as tokens.
OpenAI Codex is based on GPT-3, but also deals with source code. For source code in general, ...
- 159
1
vote
1
answer
154
views
Inference order in BERT masking task
In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order?
For example:
The ...
0
votes
1
answer
125
views
Optimal input setup for character-level text classification RNN
I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
- 101
1
vote
1
answer
187
views
Comparing Language Model of two corpora
I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
- 71
0
votes
1
answer
379
views
Feeding XLM-R embeddings to neural machine translation?
I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different
Input: He sells food.
Output (...
- 31
1
vote
1
answer
628
views
Language modelling for Spell Checker
I am working on spell checkers, I want to create a spell checker, I am confused about which model to use
Word-Level modelling
Character-Level modelling
plus I am preferring neural networks over ...
- 51
1
vote
0
answers
28
views
Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation
I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode:
...
- 767
1
vote
0
answers
100
views
Why is MLP working similar to RNN for text generation
I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus ...
- 183
1
vote
1
answer
67
views
The principle of LM deep model
Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-...
- 711
4
votes
2
answers
3k
views
how much text data is required for a meaningful use of word2vec
how much data does word2vec require? Are there any public data sets that are useful?
For example, could it be that 1000 newspaper articles are enough to use word2vec?
Here is a word2vec tutorial ...
- 141
7
votes
3
answers
4k
views
Can finite state machines be encoded as input/output for a neural network?
I want to encode finite state machines (specifically DFAs) as output (or input) of a neural network for a supervised learning task.
Are there any ways in the literature for doing this?
I've already ...
- 210
3
votes
1
answer
802
views
Neural Networks for Predictive typing
I don't have a background in neural networks. But, various studies has been proved that neural networks (feed forward / Recurrent) outperformed n-gram language modeling for predicting words in a ...
- 33
2
votes
0
answers
37
views
Importance of Random initialisation VS number of hidden units
A question crossed my mind not so long ago: I am doing experiments on Language Model with RNN (always with the same network topology: 50 hidden units, and 10M "directs connections" that are ...
- 157