3
$\begingroup$

I am using a question-and-answer dataset. My neural network takes a question and an article content, and outputs where an answer starts (as an integer). To visualize my data, how should I process it and what plot(s) should I use?

I'm considering:

Word/N-gram frequency histogram for the questions. Another one for the answers.

Plots mapping word/n-gram frequency to output features

Plots mapping word/n-gram frequencies to Shannon entropy values.

On that note, maybe using a smaller machine learning model - such as a decision tree - qnd graphing the resulting probabilities.

What is the best plot for a project like mine?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I'm not an expert but let me try to think with you. What's your vocabulary size?

I think certainly starting with a small machine learning model is a good idea, but I think that a decision tree would quickly suffer with even a medium-sized vocabulary. You would need a huge tree to do anything. So I think I would start with pretrained word embeddings, and use a small neural net to predict the starting point. This helps, because words that are close in meaning have similar vectors, and the decision tree wouldn't be able to use that kind of information.

Your suggestions for histograms don't seem bad, but you would have a histogram that is as wide as your vocabulary, which seems like it defeats the purpose of visualizing it... If you went with word embeddings, how about using a technique like UMAP to plot the questions and articles in 2D?

Improve this answer
$\endgroup$
4
  • $\begingroup$ Vocabulary size: 133,964 words $\endgroup$
    – moonman239
    May 27, 2019 at 0:48
  • $\begingroup$ That's huge. So a histogram with >100k columns isn't going to be much help. $\endgroup$
    – Paul
    May 27, 2019 at 7:58
  • $\begingroup$ I recalculated. It's actually 62,876. The previous count accidentally included punctuation marks. $\endgroup$
    – moonman239
    May 27, 2019 at 20:13
  • 2
    $\begingroup$ I still doubt the usefulness of a histogram with 60k columns. $\endgroup$
    – Paul
    May 29, 2019 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.