1 year ago

#377824

test-img

Shasu

taking spark job thread dump

I am using spark-sql-2.4.5v with java8 in my application. My application is failing due to OOM: Heap space issue

I am would like to take thread dump to analyze the GC logs. But I am not sure where and how to take the thread-dump on spark-job which is running on "yarn cluster" mode.

I added below configuration and trigger my spark-submit

--conf spark.executor.extraJavaOptions="-verbose:gc -Xloggc:hdfs::/user/xxxx/gc_log/$app_id/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \

--conf spark.driver.extraJavaOptions="-verbose:gc -Xloggc:hdfs::/user/xxxx/gc_log/$app_id/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \

But "gc.log" file not got generated.

What am I doing wrong here ? how to get "gc.log" file ?

java

apache-spark

apache-spark-sql

garbage-collection

0 Answers

Your Answer

Accepted video resources