I am training a model in Spark using Dl4J library in Yarn-Cluster mode. When I train the model on 2 lakh data (approx 200MB) then the job succeeds but when I go to train the model with 3 lakh data (approx 300MB) then the job finishes with a fail status and the model doesn't get saved in HDFS. So somehow I am reaching to a bottleneck over there.
I am using 9 containers with 1 core and 1.5GB memory to run the training job.
There is no error in yarn or spark logs.
How do I identify where the problem has occurred? How do I identify if it is the insufficient resources that is causing the problem or is the code that is the problem?