1 year ago

#327595

test-img

1stenjoydmoment

Facing issue while running MR Job on Yarn

I am trying run a MR job on Yarn. While running the MR job , the job gets stuck while cleaning up the staging dir. created by the process. below is the log snippet

2022-03-24 00:12:48 IST INFO  [defaultEventExecutorGroup-2-1] org.apache.hadoop.yarn.client.RMProxy  - Connecting to ResourceManager at localhost/127.0.0.1:8032
2022-03-24 00:12:49 IST INFO  [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input paths to process : 3
2022-03-24 00:12:49 IST INFO  [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.JobSubmitter  - number of splits:3
2022-03-24 00:12:49 IST INFO  [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.JobSubmitter  - Submitting tokens for job: job_1648055218115_0023
2022-03-24 00:12:49 IST INFO  [defaultEventExecutorGroup-2-1] org.apache.hadoop.mapreduce.JobSubmitter  - Cleaning up the staging area /user/pk/.staging/job_1648055218115_0023

Below is the dir structure on HDFS

drwx------   - pk supergroup          0 2022-03-24 00:12 /user/pk/.staging

mapred-site.xml

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:8021</value>
  </property>
<property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>

</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>

Can someone suggest how we can resolve this issue ?

hadoop

mapreduce

hadoop-yarn

0 Answers

Your Answer

Accepted video resources