[Spark] Memory model

M1-Execution memory
M2-Storage memory
M3-User memory
M4-Reserved Memory for Spark system

Note
N1-M1,M2 are flexible,
N2-Spark don’t monitor M3, if user exploit M3, might OOM.
N3-spark.memory.fraction =0.6 (default) (controls M3 / M1,M2)
N4-spark.memory.storageFraction =0.5 (default) (controls M1/M2)

Q1-UserMemory是怎样被使用的?当我们在RDD.map(function)中的function中初始化新的对象时,是在哪部分内存被初始化的?是不是就是UserMemory?
Yes!
https://www.jianshu.com/p/310fba6237d9

Q2-Runtime check spark conf
sc.getConf.getSizeAsBytes(“spark.driver.memory”)

R1-Spark memory management (Important document)
https://0x0fff.com/spark-memory-management/

R2-How does web UI calculate Storage Memory (in Executors tab)? (Really great overview)
https://stackoverflow.com/questions/43801062/how-does-web-ui-calculate-storage-memory-in-executors-tab

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s