Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Developing a application where sometimes when I make a query I get millions or 100 of thousands of records in the results.
My problem is , when I get these huge result sets, how do I visualize (in charts such as bar, radar, line graphs, etc.) this data?
Do I just take a sample of the data and visualize that? Do I just go and plot million records? What is the best way to go about this ?
Holoviews visual library can handle very large data http://holoviews.org/ http://holoviews.org/user_guide/Large_Data.html
Plotting millions of entries through histograms, pie charts, doughnut charts, tree maps, area charts, bar charts, choropleths (and so on - and on and on) does not pose any challenge. You can only find it very slow and annoying if you were to use scatter plots/violin plots, or visualise as a very large graph.
I recommend you using PCA. It finds directions which data is highly distributed in. Using this procedure, the components __ new features __ will be in descending order for the eigenvalues. Each eigenvalue that has greater value than the next eigenvalues, will have much information than them. After using PCA you can use its first three principal components for plotting. Each of the new features is a linear combination of the previous features. Using e.g. first three principal components will have so much information which will be representative of your data. In cases that data is not correlated, the preceding statement may not be always true but in your case that you have so many features, based on experience, you definitely have so many correlated features. For more information take a look at here and here which may help you.
Sampling is a totally good option, especially if the size of your data is bogging down the tool you're using to plot it.
If that's not the problem, a common issue is that plotting opaque markers will show you where data is located, but will disguise density information. For example, imagine a situation where every pixel of your plotting area is associated with at least ome observation (i.e. you have a uniformly colored plot) but one pixel is actually associated with 99% of your data. A good technique for situations like this is to try to visualize the density of the data. A simple approach is to add transparency to your markers (often by adjusting the "alpha" parameter), or you can model the density more directly with binning (e.g. a histogram or hexgrid) or with a kernel density estimate.
If you have discrete data, overplotting will likely be an issue but density might give you weird results. A good way to address this is to "jitter" your data by adding noise to one or more plotting dimensions to force your data to spread out more.
If you have time series data, you can resample to a coarser resolution: e.g., if you have a data point for every millisecond, your data will probably be easier to visualize if you aggregate by hour, day, or week.
Similarly, you can summarize the data by plotting a model. Plot $X$ vs $E[Y|X]$ instead of $X$ vs $Y$and throw in some error bands for good measure.
All that said: just try plotting it first and see what happens. Your visualization tool might do some stuff under the hood to render at least some of this manual effort unnecessary.
If you must plot raw values, use a random sampling strategy or some form of decimation (every nth value). Otherwise, computing and plotting summary statistics will be orders of magnitude faster. While lossy, careful attention to variability around the metrics will help you understand the form of the raw data
