I am using a question-and-answer dataset. My neural network takes a question and an article content, and outputs where an answer starts (as an integer). To visualize my data, how should I process it and what plot(s) should I use?

I'm considering:

Word/N-gram frequency histogram for the questions. Another one for the answers.

Plots mapping word/n-gram frequency to output features

Plots mapping word/n-gram frequencies to Shannon entropy values.

On that note, maybe using a smaller machine learning model - such as a decision tree - qnd graphing the resulting probabilities.

What is the best plot for a project like mine?


I'm not an expert but let me try to think with you. What's your vocabulary size?

I think certainly starting with a small machine learning model is a good idea, but I think that a decision tree would quickly suffer with even a medium-sized vocabulary. You would need a huge tree to do anything. So I think I would start with pretrained word embeddings, and use a small neural net to predict the starting point. This helps, because words that are close in meaning have similar vectors, and the decision tree wouldn't be able to use that kind of information.

Your suggestions for histograms don't seem bad, but you would have a histogram that is as wide as your vocabulary, which seems like it defeats the purpose of visualizing it... If you went with word embeddings, how about using a technique like UMAP to plot the questions and articles in 2D?

| improve this answer | |
  • $\begingroup$ Vocabulary size: 133,964 words $\endgroup$ – moonman239 May 27 '19 at 0:48
  • $\begingroup$ That's huge. So a histogram with >100k columns isn't going to be much help. $\endgroup$ – Paul May 27 '19 at 7:58
  • $\begingroup$ I recalculated. It's actually 62,876. The previous count accidentally included punctuation marks. $\endgroup$ – moonman239 May 27 '19 at 20:13
  • 1
    $\begingroup$ I still doubt the usefulness of a histogram with 60k columns. $\endgroup$ – Paul May 29 '19 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.