1. Flink.Storm.Sparkstreaming对比 Storm只支持流处理任务,数据是一条一条的源源不断地处理,而MapReduce.spark只支持批处理任务,spark-streaming本质上是一个批处理,采用micro-batch的方式,将数据流切分成细粒度的batch进行处理.Flink同时支持流处理和批处理,一条数据被处理完以后,序列化到缓存后,以固定的缓存块为单位进行网络数据传输,缓存块设为0为流处理,缓存块设为较大值为批处理. storm------ --------

配置环境 包括 JAVA_HOME jobmanager.rpc.address jobmanager.heap.mb 和 taskmanager.heap.mb taskmanager.numberOfTaskSlots taskmanager.tmp.dirs slaves文件 启动关闭bin/start-cluster.shbin/stop-cluster.sh   初步使用 public static void main(String[] args) throws Exception {

Flink resource manager的作用如图,   FlinkResourceManager /** * * <h1>Worker allocation steps</h1> * * <ol> * <li>The resource manager decides to request more workers. This can happen in order * to fill the initial pool, or as a result o

Apache Flink闻名已久,一直没有亲自尝试一把,这两天看了文档,发现在real-time streaming方面,Flink提供了更多高阶的实用函数. 用Apache Flink实现WordCount 下载Apache Flink 0.10.1 启动local模式 bin/start-local.sh 运行scala-shell bin/start-scala-shell.sh remote localhost 6123 Flink中JobManager的默认监听端口是6123 word

InstanceManager用于管理JobManager申请到的taskManager和slots资源 /** * Simple manager that keeps track of which TaskManager are available and alive. */ public class InstanceManager { // ------------------------------------------------------------------------ //

  参考, http://wuchong.me/blog/2016/05/25/flink-internals-window-mechanism/ http://wuchong.me/blog/2016/06/06/flink-internals-session-window/    WindowOperator window operator通过WindowAssigner和Trigger来实现它的逻辑 当一个element到达时,通过KeySelector先assign一个key,并且通过W

org.apache.flink.streaming.api.windowing.triggers;   Trigger public abstract class Trigger<T, W extends Window> implements Serializable { /** * Called for every element that gets added to a pane. The result of this will determine * whether the pane

如果要考虑易用性和效率,使用rocksDB来替代普通内存的kv是有必要的 有了rocksdb,可以range查询,可以支持columnfamily,可以各种压缩 但是rocksdb本身是一个库,是跑在RocksDBStateBackend中的 所以taskmanager挂掉后,数据还是没了, 所以RocksDBStateBackend仍然需要类似HDFS这样的分布式存储来存储snapshot   kv state需要由rockdb来管理,这是和内存或file backend最大的不同 Abstr

在Flink – Checkpoint 没有描述了整个checkpoint的流程,但是对于如何生成snapshot和恢复snapshot的过程,并没有详细描述,这里补充   StreamOperator /** * Basic interface for stream operators. Implementers would implement one of * {@link org.apache.flink.streaming.api.operators.OneInputStreamOper

  public class StreamTaskState implements Serializable, Closeable { private static final long serialVersionUID = 1L; private StateHandle<?> operatorState; private StateHandle<Serializable> functionState; private HashMap<String, KvStateSnaps

Where did we come from? With the 0.9.0-milestone1 release, Apache Flink added an API to process relational data with SQL-like expressions called the Table API. The central concept of this API is a Table, a structured data set or stream on which relat

Flink在流上最大的特点,就是引入全局snapshot,   CheckpointCoordinator 做snapshot的核心组件为, CheckpointCoordinator /** * The checkpoint coordinator coordinates the distributed snapshots of operators and state. * It triggers the checkpoint by sending the messages to the re

  Metrics是以MetricsGroup来组织的 MetricGroup MetricGroup 这就是个metric容器,里面可以放subGroup,或者各种metric 所以主要的接口就是注册, /** * A MetricGroup is a named container for {@link Metric Metrics} and further metric subgroups. * * <p>Instances of this class can be used to re

1.概述 Flink 1.1.0 版本已经在官方发布了,官方博客于 2016-08-08 更新了 Flink 1.1.0 的变动.在这 Flink 版本的发布,添加了 SQL 语法这一特性.这对于业务场景复杂,依赖于 SQL 来分析统计数据,算得上是一个不错的福利.加上之前有同学和朋友邮件中提到,Flink 官方给的示例运行有困难,能否整合一下 Flink 的案例.笔者通过本篇博客来解答一下相关疑问. 2.内容 2.1 集群部署 首先,集群的部署需要 JDK 环境.下载 JDK 以及配置 JAV

1.概述 在如今数据爆炸的时代,企业的数据量与日俱增,大数据产品层出不穷.今天给大家分享一款产品—— Apache Flink,目前,已是 Apache 顶级项目之一.那么,接下来,笔者为大家介绍Flink 的相关内容. 2.内容 2.1 What's Flink Apache Flink 是一个面向分布式数据流处理和批量数据处理的开源计算平台,它能够基于同一个Flink运行时(Flink Runtime),提供支持流处理和批处理两种类型应用的功能.现有的开源计算方案,会把流处理和批处理作为两种

https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals FLIP-1 : Fine Grained Recovery from Task Failures When a task fails during execution, Flink currently resets the entire execution graph and triggers complete re-execution f

  读accumlator JobManager 在job finish的时候会汇总accumulator的值, newJobStatus match { case JobStatus.FINISHED => try { val accumulatorResults = executionGraph.getAccumulatorsSerialized() val result = new SerializedJobExecutionResult( jobID, jobInfo.duration,

http://wuchong.me/blog/2016/04/26/flink-internals-how-to-handle-backpressure/ https://ci.apache.org/projects/flink/flink-docs-master/internals/back_pressure_monitoring.html   反压之所以会是需要的,因为当源端流量过大,而消费端无法及时消费时,可能会导致job crash:比如内存耗尽等: 其实通过合理控制资源的使用,JSto

先看例子, final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<Tuple2<Long, Long>> stream = env.addSource(...); stream .keyBy(0) .timeWindow(Time.of(2500, MILLISECONDS), Time.of(500, MILLISECONDS)

  先看 AbstractFetcher 这个可以理解就是,consumer中具体去kafka读数据的线程,一个fetcher可以同时读多个partitions的数据来看看 /** * Base class for all fetchers, which implement the connections to Kafka brokers and * pull records from Kafka partitions. * * <p>This fetcher base class imple

All transformations in Flink may look like functions (in the functional processing terminology), but are in fact stateful operators. You can make every transformation (map, filter, etc) stateful by using Flink's state interface or checkpointing insta