I am working on Rstudio on a server which has 250GB ram. But its taking too much time to handle a 2GB data file. how should i speed up my work?

  • 1
    $\begingroup$ How much time is "too much time"? 10 seconds? 10 minutes? 10 hours? What sort of "data file"? CSV? How many rows, columns? What is the rest of the spec of your machine? Are you sharing this server with other people? Does your server sysadmin limit your resources? What version of R? Post the sessionInfo() output, because if its a 32 bit R you need to switch. What are you going to do with it when you have got it? $\endgroup$ – Spacedman Jun 23 '15 at 7:31

You may or may not already know this, but here are a few basics about how R and Rstudio work and use resources.

Rstudio is a graphical user interface to R, not the interpeter/runtime environment. There is a separate "R session" that actually executes your R programs and returns results for Rstudio to display. Therefore, having Rstudio use more memory won't make any difference in the execution speed of your program.

Second, memory allocation and cleanup (a.k.a. "garbage collection") is handled automatically by the R runtime environment. "R holds objects it is using in virtual memory". Virtual memory is a combination of physical main memory and secondary storage, the mixture determined by the server OS configuration. You can use command line to change the amount of virtual memory allocated to processes. (Consult with your favorite Linux expert on this.)

Third, your speed of your program may or may not be limited by memory speed. You may be compute-bound. Do some testing to find out what is constraining performance.

Fourth, you should first ask yourself whether your program implements an efficient algorithm. Even if you aren't programming with loops and branches, the functions you call may use them, and maybe not efficiently for your application. For example, there are dramatic performance gains to be had by switching to data.table from data.frame.

Fifth, once you have chosen an efficient algorithm, you can put effort into parallel execution. The simplest way to do this is by using functions that automatically vectorize operations. A bit more complicated is to recode your program use the packages doParallel and foreach. With doParallel, you can specify the number of CPU cores to use, which on a server may range from 32 to 64 or more. Finally, if your server has a Graphics Processing Unit (GPU), it's possible for some algorithms to reprogram using the GPU commands and get massive parallelism. This option takes the most effort and has the most constraints.

| improve this answer | |
  • $\begingroup$ sixth, you could use sql $\endgroup$ – Anthony Damico Jun 23 '15 at 2:35
  • 1
    $\begingroup$ I'm not sure how any of this addresses the questioners extremely vague question. This Q should be voted down and clarification asked for in comments. $\endgroup$ – Spacedman Jun 23 '15 at 7:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.