1 year ago
#327595
1stenjoydmoment
Facing issue while running MR Job on Yarn
I am trying run a MR job on Yarn. While running the MR job , the job gets stuck while cleaning up the staging dir. created by the process. below is the log snippet
2022-03-24 00:12:48 IST INFO [defaultEventExecutorGroup-2-1] org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at localhost/127.0.0.1:8032
2022-03-24 00:12:49 IST INFO [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 3
2022-03-24 00:12:49 IST INFO [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.JobSubmitter - number of splits:3
2022-03-24 00:12:49 IST INFO [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1648055218115_0023
2022-03-24 00:12:49 IST INFO [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area /user/pk/.staging/job_1648055218115_0023
Below is the dir structure on HDFS
drwx------ - pk supergroup 0 2022-03-24 00:12 /user/pk/.staging
mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:8021</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
Can someone suggest how we can resolve this issue ?
hadoop
mapreduce
hadoop-yarn
0 Answers
Your Answer