9
$\begingroup$

I am training random forest models in R using randomForest() with 1000 trees and data frames with about 20 predictors and 600K rows. On my laptop everything works fine, but when I move to amazon ec2, to run the same thing, I get the error:

Error: cannot allocate vector of size 5.4 Gb
Execution halted

I'm using the c3.4xlarge instance type so it's pretty beefy. Does anyone know a workaround for this to get it to run on this instance? I would love to know the memory nuances that causes this problem only on the ec2 instance and not on my laptop (OS X 10.9.5 Processor 2.7 GHz Intel Core i7; Memory 16 GB 1600 MHz DDR3)

Thanks.

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
5
$\begingroup$

Here's some advice (use at your own risk!):

If the above-mentioned simpler measures don't help OR you want to achieve more scalability and/or performance, including an ability to parallelize the process on a single machine or across multiple machines, consider using bigrf R package: http://cran.r-project.org/web/packages/bigrf. Also see this discussion: https://stackoverflow.com/q/1358003/2872891.

Improve this answer
$\endgroup$
2
$\begingroup$

Additional to other ideas: reduce your data until you figure out what you can run on the Amazon instance. If it can't do 100k rows then something is very wrong, if it fails at 590k rows then its marginal.

The c3.4xlarge instance has 30Gb of RAM, so yes it should be enough.

Improve this answer
$\endgroup$
2
$\begingroup$

It is always helpful to just Google the exact error that you are seeing, excluding specifics like the actual memory of the vector. For me, the first hit was an interesting documentation called "R: Memory limits of R", where, under "Unix", one can read:

The address-space limit is system-specific: 32-bit OSes imposes a limit of no more than 4Gb: it is often 3Gb. Running 32-bit executables on a 64-bit OS will have similar limits: 64-bit executables will have an essentially infinite system-specific limit (e.g., 128Tb for Linux on x86_64 cpus).

See the OS/shell's help on commands such as limit or ulimit for how to impose limitations on the resources available to a single process. For example a bash user could use

ulimit -t 600 -v 4000000

whereas a csh user might use

limit cputime 10m limit vmemoryuse 4096m

to limit a process to 10 minutes of CPU time and (around) 4Gb of virtual memory. (There are other options to set the RAM in use, but they are not generally honoured.)

So, you should check

  1. What type of OS are you running on your EC2 instance
  2. What type of R build are you running on that OS, and make sure you run a 64bit version
  3. If both are already 64bit, then use ulimit to set memory to e.g. 8Gb: ulimit -v 8000000
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.