在docker容器下安装airflow

本人的环境是基于centos7下来安装的

一、安装docker

下载docker安装包，下载地址：https://download.docker.com/linux/static/stable/x86_64/

下载到本地后解压

tar -zxf docker-18.09..tgz

将解压出来的docker文件内容移动到 /usr/bin/ 目录下

cp docker/* /usr/bin/

将docker注册为service

新建文件

vim /etc/systemd/system/docker.service

并添加以下内容

[Unit]

Description=Docker Application Container Engine

Documentation=https://docs.docker.com

After=network-online.target firewalld.service

Wants=network-online.target

[Service]

Type=notify

# the default is not to use systemd for cgroups because the delegate issues still

# exists and systemd currently does not support the cgroup feature set required

# for containers run by docker

ExecStart=/usr/bin/dockerd

ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.

# Only systemd  and above support this version.

#TasksMax=infinity

TimeoutStartSec=

# set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

# kill only the docker process, not all processes in the cgroup

KillMode=process

# restart the docker process if it exits prematurely

Restart=on-failure

StartLimitBurst=

StartLimitInterval=60s

[Install]

WantedBy=multi-user.target

添加文件权限

chmod +x /etc/systemd/system/docker.service

systemctl daemon-reload

启动docker

systemctl start docker

验证

systemctl status docker			#查看Docker状态

docker -v			#查看Docker版本

二、安装airflow容器

先在宿主机安装mysql

上传mysql安装包

先卸载自带的mysql相关的包

 rpm -qa|grep mariadb

mariadb-libs-5.5.-.el7_5.x86_64

$ rpm -e --nodeps mariadb-libs-5.5.-.el7_5.x86_64

error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied)

$ sudo rpm -e --nodeps mariadb-libs-5.5.-.el7_5.x86_64

$ rpm -qa|grep mariadb

再依次安装

bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-common-5.7.-.el7.x86_64.rpm

warning: mysql-community-common-5.7.-.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing... ################################# [%]

Updating / installing...

:mysql-community-common-5.7.-.e################################# [%]

[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-5.7.-.el7.x86_64.rpm

warning: mysql-community-libs-5.7.-.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing... ################################# [%]

Updating / installing...

:mysql-community-libs-5.7.-.el7################################# [%]

[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-compat-5.7.-.el7.x86_64.rpm

warning: mysql-community-libs-compat-5.7.-.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing... ################################# [%]

Updating / installing...

:mysql-community-libs-compat-5.7.################################# [%]

[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-embedded-compat-5.7.-.el7.x86_64.rpm

warning: mysql-community-embedded-compat-5.7.-.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing... ################################# [%]

Updating / installing...

:mysql-community-embedded-compat-################################# [%]

[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-devel-5.7.-.el7.x86_64.rpm

warning: mysql-community-devel-5.7.-.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing... ################################# [%]

Updating / installing...

:mysql-community-devel-5.7.-.el################################# [%]

[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-client-5.7.-.el7.x86_64.rpm

warning: mysql-community-client-5.7.-.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing... ################################# [%]

Updating / installing...

:mysql-community-client-5.7.-.e################################# [%]

[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-server-5.7.-.el7.x86_64.rpm

warning: mysql-community-server-5.7.-.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing... ################################# [%]

Updating / installing...

:mysql-community-server-5.7.-.e################################# [%]

[bigdata@bigdata mysql]$

skip-grant-tables #添加这句话，这时候登入mysql就不需要密码

登录mysql，输入密码处直接回车就可以

配置msyql

mysql> flush privileges;

Query OK,  rows affected (0.00 sec)

mysql> set password for root@localhost = password('Admin123!');

Query OK,  rows affected,  warning (0.00 sec)

mysql> create user 'airflow'@'localhost' identified by 'Airflow123!';

Query OK,  rows affected (0.05 sec)

mysql> create database airflow;

Query OK, 1 row affected (0.00 sec)

mysql> create user 'airflow'@'%' identified by 'AirFlow123!';

Query OK, 0 rows affected (0.02 sec)

mysql> GRANT all privileges on airflow.* TO 'airflow'@'%'  IDENTIFIED BY 'AirFlow123!';

Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> GRANT all privileges on airflow.* TO 'root'@'172.17.0.2' IDENTIFIED BY 'Admin123!';
Query OK, 0 rows affected, 1 warning (0.04 sec)

//172.17.0.2 是我的docker容器的地址

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

修改/etc/my.cnf配置文件，在[mysqld]下添加编码配置

character_set_server=utf8

init_connect='SET NAMES utf8'

vim /etc/my.cnf

添加这一句

explicit_defaults_for_timestamp=1

编辑保存完重启mysql服务: sudo systemctl restart mysqld.service

三、docker上安装airflow

git clone https://github.com/puckel/docker-airflow.git /root/airflow //下载源码到airflow文件夹

docker run --name airflow6  -v /usr/local/airflow/dags:/usr/local/airflow/dags -v /usr/local/airflow/airflowSql:/usr/local/airflow/airflowSql  -v /home/airflow/airflow.cfg:/usr/local/airflow/airflow.cfg -d -p 8086:8080 --privileged puckel/docker-airflow   //运行容器

/* 我这里把本地的目录映射到容器里面去  通过 -v 参数进行映射**/

进入容器