I have a data set of around 3M row. I has only 2 category (category- 2:1 ratio). Now i want to visualize(scatter plot) it's distribution to understand can the data linearly separable or not(In order to choose model type).I already try this and the plot is not understandable. What will be the best way to visualize this data set?
I have three suggestions that may help.
- Reduce the point size
- Make the points highly transparent
- Downsample the points
Since you do not provide any sample data, I will use some random data to illustrate.
## The purpose of S1 is to intermix the two classes at random S1 = sample(3000000) x = c(rnorm(2000000, 0, 1), rnorm(1000000, 3,1))[S1] y = c(rnorm(2000000, 0, 1), rnorm(1000000, 3,1))[S1] z = c(rep(1,2000000), rep(2,1000000))[S1] plot(x,y, pch=20, col=rainbow(3)[z])
The base plot without any adjustments is not very nice. Let's apply suggestions 1 and 2.
plot(x,y, pch=20, cex=0.4, col=rainbow(3, alpha=0.01)[z])
Reducing the point size and making the points highly transparent helps some. This gives a better idea of the overlap between the two distributions.
If we downsample, we don't need quite as much transparency.
## The purpose of S2 is to downsample the data S2 = sample(3000000, 100000) plot(x[S2],y[S2], pch=20, cex=0.4, col=rainbow(3, alpha=0.1)[z[S2]])
This gives a different view that provides a similar, but not identical understanding of the two distributions.
These are not magic, but I think that they are helpful.
Assuming you're using Python, the
datashader module was created to effectively display very large number of points.
# import modules import pandas as pd import hvplot.pandas # read your data into dataframe (or whatever source). df = pd.read_csv('large_file.csv') # plot using hvplot; normally df.plot df.hvplot.scatter('x_column', 'y_column', datashade=True')
datashader in effect creates a series of images and only shows the data to the required resolution without over plotting. As you zoom in it updated the view with the refined detail.
If you're reading in larger than RAM datasets you may want to check out
dask as well.