Data visualisation for big data sets

Question

Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), which allow to reduce the dimension of a data set to plot it in 2 dimension. It seems fast and efficient.

In the main paper authors even provide some beautifull plots. As for exemple :

I have been unable to reproduce this kind of graphics in R with ggplot2 (computer crashing because of too many points, overlaping points...). How would one go about building a similar plot ?

Answer 1

Ggplot2 is great for simple visualisations but doesn't handle huge datasets very well, as you found out. Running out of memory is also often an issue with huge sets, and calculating prime divisibility on 30 million integers definitely counts as huge.

If you have access to more powerful systems or cloud platforms that should work a lot better. Some of the AWS and Azure etc. solutions aren't that expensive so you could try one of them.

Answer 2

The following appear to function quite smoothly and can be reparametrized easily if need be.

The key was to build a new theme to apply to the ggplot2 graph and using size options (size=1, shape=".") :

library(ggplot2)

theme_black = function(base_size = 12, base_family = "") {

  theme_grey(base_size = base_size, base_family = base_family) %+replace%

    theme(
      # Specify axis options
      axis.line = element_blank(),  
      axis.text.x = element_blank(),  
      axis.text.y = element_blank(),  
      axis.ticks = element_blank(),  
      axis.title.x = element_blank(),  
      axis.title.y = element_blank(),  
      axis.ticks.length = unit(0.3, "lines"),   
      # Specify legend options
      legend.background = element_rect(color = NA, fill = "black"),  
      legend.key = element_blank(),  
      legend.key.size = unit(1.2, "lines"),  
      legend.key.height = NULL,  
      legend.key.width = NULL,      
      legend.text = element_text(size = base_size*0.8, color = "white"),  
      legend.title = element_text(size = base_size*0.8, face = "bold", hjust = 0, color = "white"),  
      legend.position = "right",  
      legend.text.align = NULL,  
      legend.title.align = NULL,  
      legend.direction = "vertical",  
      legend.box = NULL, 
      # Specify panel options
      panel.background = element_rect(fill = "black", color  =  NA),  
      panel.border = element_blank(),  
      panel.grid.major = element_blank(),  
      panel.grid.minor = element_blank(),  
      panel.margin = unit(0.5, "lines"),   

      # Specify plot options
      plot.background = element_rect(color = "black", fill = "black"),  
      plot.title = element_text(size = base_size*1.2, color = "white"),  
      plot.margin = unit(rep(1, 4), "lines")

    )

}

So the following code with noisy colors... :

n = 100000

X1 = rnorm(n = n, 0, 1)
X2 = rnorm(n = n, 0, 1)
Y = sin(5*X1*X2+rnorm(n = n, 0, 1))


df = data.frame(X1,X2,Y)

gg <- ggplot(df, aes(x=X1, y=X2)) + 
  geom_point(aes(col=Y), size=1, shape=".") + 
  scale_color_gradient(low="blue", high="red") +
  xlim(c(-5, 5)) + 
  ylim(c(-5, 5)) + 
  labs(title="Scatterplot", 
       caption = "Source: SMAE") +
  theme_black()


plot(gg)

will give the picture below.