# Data visualisation for big data sets

Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), which allow to reduce the dimension of a data set to plot it in 2 dimension. It seems fast and efficient.

In the main paper authors even provide some beautifull plots. As for exemple :

I have been unable to reproduce this kind of graphics in R with ggplot2 (computer crashing because of too many points, overlaping points...). How would one go about building a similar plot ?

• I gave you -1, because the used software is provided in the citations. Commented Sep 19, 2019 at 8:15
• I missed that. Thanks for pointing it out. Any idea on how to perform that in R ? or where to ask ? Commented Sep 19, 2019 at 8:35

Ggplot2 is great for simple visualisations but doesn't handle huge datasets very well, as you found out. Running out of memory is also often an issue with huge sets, and calculating prime divisibility on 30 million integers definitely counts as huge.

If you have access to more powerful systems or cloud platforms that should work a lot better. Some of the AWS and Azure etc. solutions aren't that expensive so you could try one of them.

• I have access to a lot of calculation power if need be. What would be the code to obtain the image above ? Commented Sep 19, 2019 at 6:02

The following appear to function quite smoothly and can be reparametrized easily if need be.

The key was to build a new theme to apply to the ggplot2 graph and using size options (size=1, shape=".") :

library(ggplot2)

theme_black = function(base_size = 12, base_family = "") {

theme_grey(base_size = base_size, base_family = base_family) %+replace%

theme(
# Specify axis options
axis.line = element_blank(),
axis.text.x = element_blank(),
axis.text.y = element_blank(),
axis.ticks = element_blank(),
axis.title.x = element_blank(),
axis.title.y = element_blank(),
axis.ticks.length = unit(0.3, "lines"),
# Specify legend options
legend.background = element_rect(color = NA, fill = "black"),
legend.key = element_blank(),
legend.key.size = unit(1.2, "lines"),
legend.key.height = NULL,
legend.key.width = NULL,
legend.text = element_text(size = base_size*0.8, color = "white"),
legend.title = element_text(size = base_size*0.8, face = "bold", hjust = 0, color = "white"),
legend.position = "right",
legend.text.align = NULL,
legend.title.align = NULL,
legend.direction = "vertical",
legend.box = NULL,
# Specify panel options
panel.background = element_rect(fill = "black", color  =  NA),
panel.border = element_blank(),
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
panel.margin = unit(0.5, "lines"),

# Specify plot options
plot.background = element_rect(color = "black", fill = "black"),
plot.title = element_text(size = base_size*1.2, color = "white"),
plot.margin = unit(rep(1, 4), "lines")

)

}


So the following code with noisy colors... :

n = 100000

X1 = rnorm(n = n, 0, 1)
X2 = rnorm(n = n, 0, 1)
Y = sin(5*X1*X2+rnorm(n = n, 0, 1))

df = data.frame(X1,X2,Y)

gg <- ggplot(df, aes(x=X1, y=X2)) +
geom_point(aes(col=Y), size=1, shape=".") +
xlim(c(-5, 5)) +
ylim(c(-5, 5)) +
labs(title="Scatterplot",
caption = "Source: SMAE") +
theme_black()

plot(gg)


will give the picture below.