记录一次线上yarn RM频繁切换的故障

周末一大早被报警惊醒，rm频繁切换

急急忙忙排查看到两处错误日志

错误信息1

ervation <memory:0, vCores:0>

2019-12-21 11:51:57,781 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type APP_ATTEMPT_REMOVED to the scheduler

java.lang.NullPointerException

    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.unreserveResource(FSSchedulerNode.java:88)

    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.unreserve(FSAppAttempt.java:589)

    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.completedContainerInternal(FairScheduler.java:899)

    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:564)

    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.removeApplicationAttempt(FairScheduler.java:846)

    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1479)

    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:117)

    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:804)

    at java.lang.Thread.run(Thread.java:748)

错误信息2

明月照我去搬砖 2019/12/21 14:51:07

2019-12-21 07:37:45,533 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type APP_ATTEMPT_REMOVED to the scheduler

java.lang.NullPointerException

        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.completedContainerInternal(FairScheduler.java:902)

        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:564)

        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.removeApplicationAttempt(FairScheduler.java:837)

        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1475)

        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:117)

        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:804)

        at java.lang.Thread.run(Thread.java:748)

2019-12-21 07:37:45,534 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye..

查看源码处FairScheduler

 @Override

  protected void completedContainerInternal(

      RMContainer rmContainer, ContainerStatus containerStatus,

      RMContainerEventType event) {

    try {

      writeLock.lock();

      Container container = rmContainer.getContainer();

      // Get the application for the finished container

      FSAppAttempt application =

        getCurrentAttemptForContainer(container.getId());

      ApplicationId appId =

        container.getId().getApplicationAttemptId().getApplicationId();

      if (application == null) {

        LOG.info("Container " + container + " of" +

          " finished application " + appId +

          " completed with event " + event);

        return;

      }

      // Get the node on which the container was allocated

      FSSchedulerNode node = getFSSchedulerNode(container.getNodeId());

      if (rmContainer.getState() == RMContainerState.RESERVED) {

        application.unreserve(rmContainer.getReservedPriority(), node); //这里将node上该container资源释放

      } else {

        try {

          application.containerCompleted(rmContainer, containerStatus, event);

          node.releaseContainer(rmContainer.getContainerId(), false);

          updateRootQueueMetrics();

          LOG.info("Application attempt " + application.getApplicationAttemptId()

                  + " released container " + container.getId() + " on node: " + node

                  + " with event: " + event);

        }catch (Exception e){

          LOG.error(e.getMessage(), e);

        }

      }

    } finally {

      writeLock.unlock();

    }

  }

跟进去看下

  /**

   * Remove the reservation on {@code node} at the given {@link Priority}.

   * This dispatches SchedulerNode handlers as well.

   */

  public void unreserve(Priority priority, FSSchedulerNode node) {

    RMContainer rmContainer = node.getReservedContainer();

    unreserveInternal(priority, node);

    node.unreserveResource(this);

    clearReservation(node);

    getMetrics().unreserveResource(node.getPartition(),

        getUser(), rmContainer.getContainer().getResource());

  }

  @Override

  public synchronized void unreserveResource(

      SchedulerApplicationAttempt application) {

    // Cannot unreserve for wrong application...

    ApplicationAttemptId reservedApplication =

        getReservedContainer().getContainer().getId().getApplicationAttemptId(); //获取不到该container的attemptId 报空指针

    if (!reservedApplication.equals(

        application.getApplicationAttemptId())) {

      throw new IllegalStateException("Trying to unreserve " +

          " for application " + application.getApplicationId() +

          " when currently reserved " +

          " for application " + reservedApplication.getApplicationId() +

          " on node " + this);

    }

    setReservedContainer(null);

    this.reservedAppSchedulable = null;

  }

第二处报错是

rmContainer为null 了对removeapplicationattent的调用和对相同尝试的moveApplication的处理顺序很短则应用程序尝试仍将包含队列引用，
但已从队列的应用程序列表中删除如果对removeapplicationattent的两个调用连续出现，则应用程序仍将包含队列引用，但已从队列的应用程序列表
中删除在这两种情况下，第二个调用必须在进行removeApplication调
用之前进入。

其实就是重复释放container 但container已经在该节点上释放了 有一个状态不一致问题
这边是用的写锁 当一个线程已经读到containerId 另一线程释放掉 再次释放 就会出现异常

修改方法一

 /**

   * Clean up a completed container.

   */

  @Override

  protected synchronized void completedContainerInternal(

      RMContainer rmContainer, ContainerStatus containerStatus,

      RMContainerEventType event) {

    try {

     // writeLock.lock();//注释写锁 改用重锁

      Container container = rmContainer.getContainer();

      // Get the application for the finished container

      FSAppAttempt application =

        getCurrentAttemptForContainer(container.getId());

      ApplicationId appId =

        container.getId().getApplicationAttemptId().getApplicationId();

      if (application == null) {

        LOG.info("Container " + container + " of" +

          " finished application " + appId +

          " completed with event " + event);

        return;

      }

修改方法二

// Get the node on which the container was allocated

      FSSchedulerNode node = getFSSchedulerNode(container.getNodeId());

      try {

      if (rmContainer.getState() == RMContainerState.RESERVED) {

        application.unreserve(rmContainer.getReservedPriority(), node);

      } else {

       // try {  //将try移到上方  覆盖unreserve方法

  application.containerCompleted(rmContainer, containerStatus, event);
 node.releaseContainer(rmContainer.getContainerId(), false);
 updateRootQueueMetrics(); 
LOG.info("Application attempt " + application.getApplicationAttemptId() + " released container " + container.getId(
) + " on node: " + node + " with event: " + event);
 }catch (Exception e){
 LOG.error(e.getMessage(), e); //将该异常处理掉而不是抛出
} }

巴特西

记录一次线上yarn RM频繁切换的故障

最新文章

热门文章