# When does cache get expired for a RDD in pyspark?

We use .cache() on RDD for persistent caching of an dataset, My concern is when this cached will be expired?.

dt = sc.parallelize([2, 3, 4, 5, 6])
dt.cache()


It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the pipeline and put it in cache again. If this would be too expensive, unpersist other RDDs, don't cache them in the first place or persist them on your file system.

In addition to Jan's answer, I would like to point out that serialized RDD storage(/caching) works much better than normal RDD caching for large datasets.

It also helps optimize garbage collection, in case of large datasets.

• Just a note: MEMORY_ONLY_SER is only available in Scala/Java, not Python. – Def_Os Feb 3 '17 at 5:15
From the terminal, you can use rdd.unpersist() or sqlContext.uncacheTable("sparktable") to remove the RDD or tables from Memory. Spark made for Lazy Evaluation, unless and until you say any action, it does not load or process any data into the RDD or DataFrame.