Flink（三） —— 运行架构
Flink运行时组件
- JobManager 作业管理器
- TaskManager 任务管理器
- ResourceManager 资源管理器
- Dispatcher 分发器
任务提交流程
任务调度原理
Job Managers, Task Managers, Clients
The Flink runtime consists of two types of processes:
The JobManagers (also called masters) coordinate the distributed execution. They schedule tasks, coordinate checkpoints, coordinate recovery on failures, etc.
There is always at least one Job Manager. A high-availability setup will have multiple JobManagers, one of which one is always the leader, and the others are standby.
The TaskManagers (also called workers) execute the tasks (or more specifically, the subtasks) of a dataflow, and buffer and exchange the data streams.
There must always be at least one TaskManager.
The JobManagers and TaskManagers can be started in various ways: directly on the machines as a standalone cluster, in containers, or managed by resource frameworks like YARN or Mesos. TaskManagers connect to JobManagers, announcing themselves as available, and are assigned work.
The client is not part of the runtime and program execution, but is used to prepare and send a dataflow to the JobManager. After that, the client can disconnect, or stay connected to receive progress reports. The client runs either as part of the Java/Scala program that triggers the execution, or in the command line process ./bin/flink run ....
Task Slots and Resources
Each worker (TaskManager) is a JVM process, and may execute one or more subtasks in separate threads. To control how many tasks a worker accepts, a worker has so called task slots (at least one).
每个TaskManager都是一个JVM进程，可以在独立的线程中执行一个或多个子任务。TaskManager有Task Slots来控制可以接收多少个任务（一个TaskManager至少有一个Task Slot）。
Each task slot represents a fixed subset of resources of the TaskManager. A TaskManager with three slots, for example, will dedicate 1/3 of its managed memory to each slot. Slotting the resources means that a subtask will not compete with subtasks from other jobs for managed memory, but instead has a certain amount of reserved managed memory. Note that no CPU isolation happens here; currently slots only separate the managed memory of tasks.
By adjusting the number of task slots, users can define how subtasks are isolated from each other. Having one slot per TaskManager means each task group runs in a separate JVM (which can be started in a separate container, for example). Having multiple slots means more subtasks share the same JVM. Tasks in the same JVM share TCP connections (via multiplexing) and heartbeat messages. They may also share data sets and data structures, thus reducing the per-task overhead.
参考文档
Flink Distributed Runtime Environment
最新文章
- 使用 Spring 进行单元测试
- 论checkbox和radio的样式美化问题
- Android AChartEngine 去除折线图黑边
- Celery 和 Redis 入门
- IOS Note - Outlet(插座) & Action(动作)
- POJ-1118（超时，但未找到原因）
- ubuntu12.04下 安装虚拟主机
- ESFramework 4.0 快速上手（06） －－ Rapid引擎（续）
- Python [习题] 字典排序
- 获取 Let's Encrypt 免费通配符证书实现Https
- ●洛谷P2495 [SDOI2011]消耗战
- 原生js实例对象方法
- kafka性能调优(转)
- DataGridView获取或者设置当前单元格的内容
- onSaveInstanceState场景
- axure元件库导入后重启程序元件库消失问题
- Java与openssl的RSA算法
- 什么是虚拟DOM?为啥虚拟DOM可以提升性能？
- LCX端口内网映射转发
- NS3 利用Gnuplot生成拥塞窗口例子fifth.cc的png图像