iSCSI网关集成了Ceph存储和iSCSI标准，以提供一个高可用性(HA) iSCSI目标，该目标将RADOS块设备(RBD)映像导出为SCSI磁盘。iSCSI协议允许客户机 (initiator) 通过TCP/IP网络向SCSI存储设备( targets )发送SCSI命令。这允许异构客户机访问Ceph存储集群。

每个iSCSI网关运行Linux IO目标内核子系统(LIO)，以提供iSCSI协议支持。LIO利用用户空间通过( TCMU ) 与Ceph的librbd库交互，并向iSCSI客户机暴露RBD镜像。使用Ceph的iSCSI网关，可以有效地运行一个完全集成的块存储基础设施，它具有传统存储区域网络(SAN)的所有特性和优点。

RBD 作为 VMware ESXI datastore 是否支持？

（1）目前来说，RBD是不支持datastore形式。

（2）iSCSI 是支持 datastore 这种方式，可以作为VMware Esxi 虚拟机提供存储功能，性价比非常不错的选择。

1、创建池和镜像

（1）创建池

# ceph osd pool create iscsi-images   replicated

# ceph osd pool application enable iscsi-images rbd

（2）创建images

# rbd --pool iscsi-images create --size= 'iscsi-gateway-image001'

# rbd --pool iscsi-images create --size= 'iscsi-gateway-image002'

# rbd --pool iscsi-images create --size= 'iscsi-gateway-image003'

# rbd --pool iscsi-images create --size= 'iscsi-gateway-image004'

（3）显示images

# rbd ls -p iscsi-images

iscsi-gateway-image001

iscsi-gateway-image002

iscsi-gateway-image003

iscsi-gateway-image004

2、deepsea 方式安装iSCSI网关

（1）node001 和 node002节点上安装，编辑policy.cfg 文件

vim /srv/pillar/ceph/proposals/policy.cfg

  ......

# IGW

role-igw/cluster/node00[-]*.sls

  ......

（2）运行 stage 2 和 stage 4

# salt-run state.orch ceph.stage.

# salt 'node001*' pillar.items

    public_network:

        192.168.2.0/

    roles:

        - mon

        - mgr

        - storage

        - igw

    time_server:

        admin.example.com

# salt-run state.orch ceph.stage.

3、手动方式安装iSCSI网关

（1）node003 节点安装 iscsi 软件包

# zypper -n in -t pattern ceph_iscsi

# zypper -n in tcmu-runner  tcmu-runner-handler-rbd \

  ceph-iscsi patterns-ses-ceph_iscsi python3-Flask python3-click python3-configshell-fb \

  python3-itsdangerous python3-netifaces python3-rtslib-fb \

  python3-targetcli-fb python3-urwid targetcli-fb-common

（2）admin节点创建key,并复制到 node003

# ceph auth add client.igw.node003 mon 'allow *' osd 'allow *' mgr 'allow r'

# ceph auth get client.igw.node003

client.igw.node003

        key: AQC0eotdAAAAABAASZrZH9KEo0V0WtFTCW9AHQ==

        caps: [mgr] allow r

        caps: [mon] allow *

        caps: [osd] allow *
# ceph auth get client.igw.node003 >> /etc/ceph/ceph.client.igw.node003.keyring

# scp /etc/ceph/ceph.client.igw.node003.keyring node003:/etc/ceph

（3）node003 节点启动服务

# systemctl start tcmu-runner.service

# systemctl enable tcmu-runner.service

（4）node003 节点创建配置文件

# vim /etc/ceph/iscsi-gateway.cfg

[config]

cluster_client_name = client.igw.node003

pool = iscsi-images

trusted_ip_list = 192.168.2.42,192.168.2.40,192.168.2.41

minimum_gateways =

fqdn_enabled=true

# Additional API configuration options are as follows, defaults shown.

api_port =

api_user = admin

api_password = admin

api_secure = false

# Log level

logger_level = WARNING

（5）启动 RBD target 服务

# systemctl start rbd-target-api.service

# systemctl enable rbd-target-api.service

（6）显示配置信息

# gwcli info

HTTP mode          : http

Rest API port      :

Local endpoint     : http://localhost:5000/api

Local Ceph Cluster : ceph

2ndary API IP's    : 192.168.2.42,192.168.2.40,192.168.2.41
# gwcli ls

o- / ...................................................................... [...]

  o- cluster ...................................................... [Clusters: ]

  | o- ceph ......................................................... [HEALTH_OK]

  |   o- pools ....................................................... [Pools: ]

  |   | o- iscsi-images ........ [(x3), Commit: .00Y/15718656K (%), Used: 192K]

  |   o- topology ............................................. [OSDs: ,MONs: ]

  o- disks .................................................... [.00Y, Disks: ]

  o- iscsi-targets ............................ [DiscoveryAuth: None, Targets: ]

4、Dashboard 添加 iscsi 网关

（1）Admin节点上，查看 dashboard iSCSI 网关

admin:~ # ceph dashboard iscsi-gateway-list

{"gateways": {"node002.example.com": {"service_url": "http://admin:admin@192.168.2.41:5000"},

 "node001.example.com": {"service_url": "http://admin:admin@192.168.2.40:5000"}}}

（2）添加 iSCSI 网关

# ceph dashboard iscsi-gateway-add http://admin:admin@192.168.2.42:5000

# ceph dashboard iscsi-gateway-list

{"gateways": {"node002.example.com": {"service_url": "http://admin:admin@192.168.2.41:5000"},

 "node001.example.com": {"service_url": "http://admin:admin@192.168.2.40:5000"},

 "node003.example.com": {"service_url": "http://admin:admin@192.168.2.42:5000"}}}

（3）登陆 Dashboard 查看 iSCSI 网关

5、Export RBD Images via iSCSI

（1）创建 iSCSI target name

# gwcli

gwcli > /> cd /iscsi-targets

gwcli > /iscsi-targets> create iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01

（2）添加 iSCSI 网关

gwcli > /iscsi-targets> cd iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/gateways

/iscsi-target...tvol/gateways> create node001.example.com 172.200.50.40

/iscsi-target...tvol/gateways> create node002.example.com 172.200.50.41

/iscsi-target...tvol/gateways> create node003.example.com 172.200.50.42

/iscsi-target...ay01/gateways> ls

o- gateways ......................................................... [Up: 3/3, Portals: 3]

  o- node001.example.com ............................................. [172.200.50.40 (UP)]

  o- node002.example.com ............................................. [172.200.50.41 (UP)]

  o- node003.example.com ............................................. [172.200.50.42 (UP)]

注意：安装主机名来定义

/iscsi-target...tvol/gateways> create node002 172.200.50.41

The first gateway defined must be the local machine

（3）添加 RBD 镜像

/iscsi-target...tvol/gateways> cd /disks

/disks> attach iscsi-images/iscsi-gateway-image001

/disks> attach iscsi-images/iscsi-gateway-image002

（4）target 和 RBD 镜像建立映射关系

/disks> cd /iscsi-targets/iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/disks

/iscsi-target...teway01/disks> add iscsi-images/iscsi-gateway-image001

/iscsi-target...teway01/disks> add iscsi-images/iscsi-gateway-image002

（5）设置不验证

gwcli > /> cd /iscsi-targets/iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/hosts

/iscsi-target...teway01/hosts> auth disable_acl

/iscsi-target...teway01/hosts> exit

（6）查看配置信息

node001:~ # gwcli ls

o- / ............................................................................... [...]

  o- cluster ............................................................... [Clusters: 1]

  | o- ceph .................................................................. [HEALTH_OK]

  |   o- pools ................................................................ [Pools: 1]

  |   | o- iscsi-images .................. [(x3), Commit: 6G/15717248K (40%), Used: 1152K]

  |   o- topology ...................................................... [OSDs: 6,MONs: 3]

  o- disks ................................................................ [6G, Disks: 2]

  | o- iscsi-images .................................................. [iscsi-images (6G)]

  |   o- iscsi-gateway-image001 ............... [iscsi-images/iscsi-gateway-image001 (2G)]

  |   o- iscsi-gateway-image002 ............... [iscsi-images/iscsi-gateway-image002 (4G)]

  o- iscsi-targets ..................................... [DiscoveryAuth: None, Targets: 1]

    o- iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01 .............. [Gateways: 3]

      o- disks ................................................................ [Disks: 2]

      | o- iscsi-images/iscsi-gateway-image001 .............. [Owner: node001.example.com]

      | o- iscsi-images/iscsi-gateway-image002 .............. [Owner: node002.example.com]

      o- gateways .................................................. [Up: 3/3, Portals: 3]

      | o- node001.example.com ...................................... [172.200.50.40 (UP)]

      | o- node002.example.com ...................................... [172.200.50.41 (UP)]

      | o- node003.example.com ...................................... [172.200.50.42 (UP)]

      o- host-groups ........................................................ [Groups : 0]

      o- hosts .................................................... [Hosts: 0: Auth: None]

6、使用 Dashboard 界面输出 RBD Images

（1）添加 iSCSI target

（2）编写 target IQN，并且添加镜像 Portals 和 images

（3）查看新添加 iSCSI target 信息

7、Linux 客户端访问

（1）启动 iscsid 服务

  • SLES or RHEL
# systemctl start iscsid.service

# systemctl enable iscsid.service
  • Debian or Ubuntu
# systemctl start open-iscsi

（2）发现和连接 targets

# iscsiadm -m discovery -t st -p 172.200.50.40

172.200.50.40:, iqn.-.com.suse-iscsi.iscsi01.x86:iscsi-gateway01

172.200.50.41:, iqn.-.com.suse-iscsi.iscsi01.x86:iscsi-gateway01

172.200.50.42:, iqn.-.com.suse-iscsi.iscsi01.x86:iscsi-gateway01

172.200.50.40:, iqn.-.com.suse-iscsi.iscsi01.x86:iscsi-gateway02

172.200.50.41:, iqn.-.com.suse-iscsi.iscsi01.x86:iscsi-gateway02

172.200.50.42:, iqn.-.com.suse-iscsi.iscsi01.x86:iscsi-gateway02

（3）登录target

# iscsiadm -m node -p 172.200.50.40 --login

# iscsiadm -m node -p 172.200.50.41 --login

# iscsiadm -m node -p 172.200.50.42 --login
# lsblk

NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

sda               :        25G   disk

├─sda1            :       509M   part /boot

└─sda2            :      .5G   part

  ├─vg00-lvswap :         2G   lvm  [SWAP]

  └─vg00-lvroot :     .5G   lvm  /

sdb               :      100G   disk

└─vg00-lvroot   :     .5G   lvm  /

sdc               :        2G   disk

sdd               :        2G   disk

sde               :        4G   disk

sdf               :        4G   disk

sdg               :        2G   disk

sdh               :       4G   disk

sdi               :       2G   disk

sdj               :       4G   disk

sdk               :       2G   disk

sdl               :       2G   disk

sdm               :       4G   disk

sdn               :       4G   disk

（4）如果系统上已安装 lsscsi 实用程序，您可以使用它来枚举系统上可用的 SCSI 设备：

# lsscsi

[:::]    cd/dvd  NECVMWar VMware SATA CD01 1.00  /dev/sr0

[:::]   disk    VMware,  VMware Virtual S 1.0   /dev/sda

[:::]   disk    VMware,  VMware Virtual S 1.0   /dev/sdb

[:::]   disk    SUSE     RBD              4.0   /dev/sdc

[:::]   disk    SUSE     RBD              4.0   /dev/sde

[:::]   disk    SUSE     RBD              4.0   /dev/sdd

[:::]   disk    SUSE     RBD              4.0   /dev/sdf

[:::]   disk    SUSE     RBD              4.0   /dev/sdg

[:::]   disk    SUSE     RBD              4.0   /dev/sdh

[:::]   disk    SUSE     RBD              4.0   /dev/sdi

[:::]   disk    SUSE     RBD              4.0   /dev/sdj

[:::]   disk    SUSE     RBD              4.0   /dev/sdk

[:::]   disk    SUSE     RBD              4.0   /dev/sdm

[:::]   disk    SUSE     RBD              4.0   /dev/sdl

[:::]   disk    SUSE     RBD              4.0   /dev/sdn

（5）多路径设置

# zypper in multipath-tools

# modprobe dm-multipath path

# systemctl start multipathd.service

# systemctl enable multipathd.service
# multipath -ll

36001405863b0b3975c54c5f8d1ce0e01 dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio=50 status=active

| `- ::: sdh : active ready running  <=== 单条链路 active

`-+- policy='service-time 0' prio= status=enabled

  |- ::: sde :  active ready running

  `- ::: sdm : active ready running

3600140529260bf41c294075beede0c21 dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio= status=active

| `- ::: sdc :  active ready running

`-+- policy='service-time 0' prio= status=enabled

  |- ::: sdg :  active ready running

  `- ::: sdk : active ready running

360014055d00387c82104d338e81589cb dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio= status=active

| `- ::: sdl : active ready running

`-+- policy='service-time 0' prio= status=enabled

  |- ::: sdd :  active ready running

  `- ::: sdi : active ready running

3600140522ec3f9612b64b45aa3e72d9c dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio= status=active

| `- ::: sdf :  active ready running

`-+- policy='service-time 0' prio= status=enabled

  |- ::: sdj : active ready running

  `- ::: sdn : active ready running

（5）编辑多路径配置文件

# vim /etc/multipath.conf

defaults {

    user_friendly_names yes

}

devices {

    device {

        vendor "(LIO-ORG|SUSE)"

        product "RBD"

        path_grouping_policy "multibus" # 所有有效路径在一个优先组群中

        path_checker "tur"              # 在设备中执行 TEST UNIT READY 命令。

        features ""

        hardware_handler "1 alua"       # 在切换路径组群或者处理 I/O 错误时用来执行硬件具体动作的模块。

        prio "alua"

        failback "immediate"

        rr_weight "uniform"             # 所有路径都有相同的加权

        no_path_retry                 # 路径故障后，重试12次，每次5秒

        rr_min_io                    # 指定切换到当前路径组的下一个路径前路由到该路径的 I/O 请求数。

    }

}
# systemctl stop multipathd.service

# systemctl start multipathd.service

（6）查看多路径状态

# multipath -ll

mpathd (3600140522ec3f9612b64b45aa3e72d9c) dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

`-+- policy='service-time 0' prio= status=active

  |- ::: sdf :  active ready running  <=== 多条链路 active

  |- ::: sdj : active ready running

  `- ::: sdn : active ready running

mpathc (360014055d00387c82104d338e81589cb) dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

`-+- policy='service-time 0' prio= status=active

  |- ::: sdd :  active ready running

  |- ::: sdi : active ready running

  `- ::: sdl : active ready running

mpathb (36001405863b0b3975c54c5f8d1ce0e01) dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

`-+- policy='service-time 0' prio= status=active

  |- ::: sde :  active ready running

  |- ::: sdh : active ready running

  `- ::: sdm : active ready running

mpatha (3600140529260bf41c294075beede0c21) dm- SUSE,RBD

size=.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

`-+- policy='service-time 0' prio= status=active

  |- ::: sdc :  active ready running

  |- ::: sdg :  active ready running

  `- ::: sdk : active ready running

（7）显示当前的device mapper的信息

# dmsetup ls --tree

mpathd (:)

 ├─ (:)

 ├─ (:)

 └─ (:)

mpathc (:)

 ├─ (:)

 ├─ (:)

 └─ (:)

mpathb (:)

 ├─ (:)

 ├─ (:)

 └─ (:)

mpatha (:)

 ├─ (:)

 ├─ (:)

 └─ (:)

vg00-lvswap (:)

 └─ (:)

vg00-lvroot (:)

 ├─ (:)

 └─ (:)

（8）客户端 yast iscsi-client 工具查看

iSCSI的其他常用操作（客户端）

（1）列出所有target

# iscsiadm -m node

（2）连接所有target

# iscsiadm -m node -L all

（3）连接指定target

# iscsiadm -m node -T iqn.... -p 172.29.88.62 --login

（4）使用如下命令可以查看配置信息

# iscsiadm -m node -o show -T iqn.-.com.synology:rackstation.exservice-bak

（5）查看目前 iSCSI target 连接状态

# iscsiadm -m session

# iscsiadm: No active sessions.

(目前没有已连接的 iSCSI target)
（6）断开所有target

# iscsiadm -m node -U all

（7）断开指定target

# iscsiadm -m node -T iqn... -p 172.29.88.62 --logout

（8）删除所有node信息

# iscsiadm -m node --op delete

（9）删除指定节点（/var/lib/iscsi/nodes目录下，先断开session）

# iscsiadm -m node -o delete -name iqn.-.cn.nayun:test-

（10）删除一个目标（/var/lib/iscsi/send_targets目录下）

# iscsiadm --mode discovery -o delete -p 172.29.88.62:

最新文章

  1. ASIHTTPRequest类库简介和使用说明
  2. numpy——linspace创建等差数列
  3. 001_JavaScript 错误 - Throw、Try 和 Catch
  4. Asp.net Core WebApi 支持json/xml格式的数据返回
  5. C语言 单引号 双引号问题
  6. hive数据导入方法
  7. android屏蔽软键盘并且显示光标
  8. 完全参照系统自带的DatePickerDialog、TimePickerDialog的源代码仿写的DateTimePickerDialog
  9. cadence异型孔
  10. Java基础知识强化之网络编程笔记25：Android网络通信之 Future接口介绍（Java程序执行超时）
  11. 在Windows下用MingW 4.5.2编译live555
  12. 正&quot;/&quot; 和 反&quot;\&quot;的区别？
  13. 让微信，qq，uc浏览器使用全屏模式，全屏模式里，浏览器是不会上下左右滑动出现背景的
  14. Python 3.3.2 round函数并非&quot;四舍五入&quot;
  15. 国内第一本micropython的书出版《机器人Python极客编程入门与实战》
  16. vue生命週期
  17. React Mixins
  18. Java 条件编译
  19. golang的日志系统log和glog
  20. cas 服务端认证流程

热门文章

  1. 2019DX#1
  2. codeforces 813 D. Two Melodies（dp）
  3. Robot Framework自动化测试
  4. Python---列表的学习（二）
  5. moco 入门及问题解决
  6. 实现一个简易版的vuex持久化工具
  7. Helm 从入门到实践 | 从 0 开始制作一个 Helm Charts
  8. MySQL索引原理及SQL优化
  9. 去掉Myeclipse对JS等文件的验证
  10. 泛型接口、JAVA API、包装类