1 year ago
#377824
Shasu
taking spark job thread dump
I am using spark-sql-2.4.5v with java8 in my application. My application is failing due to OOM: Heap space issue
I am would like to take thread dump to analyze the GC logs. But I am not sure where and how to take the thread-dump on spark-job which is running on "yarn cluster" mode.
I added below configuration and trigger my spark-submit
--conf spark.executor.extraJavaOptions="-verbose:gc -Xloggc:hdfs::/user/xxxx/gc_log/$app_id/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
--conf spark.driver.extraJavaOptions="-verbose:gc -Xloggc:hdfs::/user/xxxx/gc_log/$app_id/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
But "gc.log" file not got generated.
What am I doing wrong here ? how to get "gc.log" file ?
java
apache-spark
apache-spark-sql
garbage-collection
0 Answers
Your Answer