Ceph 块存储
任何普通的linux主机都可以充当ceph客户机,客户机通过网络与ceph存储集群交互以存储或检索用户数据。Ceph RBD支持已经添加到linux主线内核中,从2.6.34以及以后版本开始。
===================在管理端192.168.20.181操作================================
su - ceph-admin
cd my-cluster
创建ceph块客户端用户名和认证密钥
ceph auth get-or-create client.rbd mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=rbd' |tee ./ceph.client.rbd.keyring
===================在客户端192.168.20.184操作================================
修改hosts文件:
cat /etc/hosts
192.168.20.181 c720181
192.168.20.182 c720182
192.168.20.183 c720183
192.168.20.184 c720184
创建存放密钥和配置文件目录:
mkdir -p /etc/ceph
===================在管理端192.168.20.181操作================================
把密钥和配置文件拷贝到客户端
scp ceph.client.rbd.keyring /etc/ceph/ceph.conf root@192.168.20.184:/etc/ceph/
===================在客户端192.168.20.184操作================================
检查客户端是否符合块设备环境要求
[root@c720184 ~]# uname -r
3.10.0-957.el7.x86_64 #高于2.6.34就可以
[root@c720184 ~]# modprobe rbd
[root@c720184 ~]# echo $?
0 #说明执行成功
安装ceph客户端
yum -y install ceph (如果没有配置ceph.repo 和 epel.repo 要先配置,否则安装不成功)
连接ceph集群:
ceph -s --name client.rbd
(1)集群创建存储池
默认创建块设备,会直接创建在rbd池中,但使用ceph-deploy安装(L版ceph)后,该rbd池并没有创建
#创建池
===================在管理端192.168.20.181操作================================
[ceph-admin@c720181 my-cluster]$ ceph osd lspools #查看集群池存储池
[ceph-admin@c720181 my-cluster]$ ceph osd pool create rbd 128 #128为place group 数量
pool 'rbd' created
[ceph-admin@c720181 my-cluster]$ ceph osd lspools
1 rbd,
(2)客户端创建块设备
[root@c720184 ~]# rbd create rbd1 --size 10240 --name client.rbd
[root@c720184 ~]# rbd ls -p rbd --name client.rbd
rbd1
[root@c720184 ~]# rbd list --name client.rbd
rbd1
[root@c720184 ~]# rbd --image rbd1 info --name client.rbd
rbd image 'rbd1':
size 10GiB in 2560 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.106b6b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
create_timestamp: Sun Aug 18 18:56:02 2019
(3)客户端映射块设备
[root@c720184 ~]# rbd map --image rbd1 --name client.rbd
直接映射会报如下错误,是因为创建的块设备属性对于该内核版本不支持。
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
解决方式有如下三种:
a.动态禁用 (建议)
rbd feature disable rbd1 exclusive-lock object-map deep-flatten fast-diff --name client.rbd
b.创建RBD块设备时,只启用分层特性
rbd create rbd1 --size 10240 --image-feature layering --name client.rbd
c.ceph配置文件中禁用
rbd_default_features = 1
比如这里使用动态禁用:(截图里面少了--name client.rbd,不然会报错)
rbd feature disable rbd1 exclusive-lock object-map deep-flatten fast-diff --name client.rbd
重新映射块设备:
[root@c720184 ~]# rbd map --image rbd1 --name client.rbd
(4)创建文件系统并挂载
[root@c720184 ~]# fdisk -l /dev/rbd0
Disk /dev/rbd0: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
[root@c720184 ~]# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0 isize=512 agcount=16, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@c720184 ~]# mkdir /mnt/ceph-disk1
[root@c720184 ~]# mount /dev/rbd0 /mnt/ceph-disk1
[root@c720184 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 17G 1.5G 16G 9% /
devtmpfs 908M 0 908M 0% /dev
tmpfs 920M 0 920M 0% /dev/shm
tmpfs 920M 8.5M 911M 1% /run
tmpfs 920M 0 920M 0% /sys/fs/cgroup
/dev/sda1 1014M 145M 870M 15% /boot
tmpfs 184M 0 184M 0% /run/user/0
/dev/rbd0 10G 33M 10G 1% /mnt/ceph-disk1
(5)写入数据测试
[root@c720184 ~]# dd if=/dev/zero of=/mnt/ceph-disk1/file1 count=100 bs=1M
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.04015 s, 101 MB/s
[root@c720184 ~]# ll -h /mnt/ceph-disk1/
total 100M
-rw-r--r--. 1 root root 100M Aug 18 19:11 file1
(6)做成服务,设置开机自动挂载
wget -O /usr/local/bin/rbd-mount https://raw.githubusercontent.com/aishangwei/ceph-demo/master/client/rbd-mount
#注意需要修改里面的配置参数
#!/bin/bash
# Pool name where block device image is stored
export poolname=rbd
# Disk image name
export rbdimage=rbd1
# Mounted Directory
export mountpoint=/mnt/ceph-disk1
# Image mount/unmount and pool are passed from the systemd service as arguments
# Are we are mounting or unmounting
if [ "$1" == "m" ]; then
modprobe rbd
rbd feature disable $rbdimage object-map fast-diff deep-flatten
rbd map $rbdimage --id rbd --keyring /etc/ceph/ceph.client.rbd.keyring
mkdir -p $mountpoint
mount /dev/rbd/$poolname/$rbdimage $mountpoint
fi
if [ "$1" == "u" ]; then
umount $mountpoint
rbd unmap /dev/rbd/$poolname/$rbdimage
fi
#添加可执行权限
chmod +x /usr/local/bin/rbd-mount
wget -O /etc/systemd/system/rbd-mount.service https://raw.githubusercontent.com/aishangwei/ceph-demo/master/client/rbd-mount.service
[root@c720184 ~]# systemctl daemon-reload
[root@c720184 ~]# systemctl enable rbd-mount.service
测试重启是否自动挂载:
reboot -f
df -h
这里为了节省时间,没有重启,而是先取消挂载,启动挂载服务看是否挂载上。
[root@c720184 ~]# umount /mnt/ceph-disk1/
[root@c720184 ~]# systemctl start rbd-mount.service
[root@client ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl_-root 50G 1.8G 49G 4% /
devtmpfs 910M 0 910M 0% /dev
tmpfs 920M 0 920M 0% /dev/shm
tmpfs 920M 17M 904M 2% /run
tmpfs 920M 0 920M 0% /sys/fs/cgroup
/dev/sda1 1014M 184M 831M 19% /boot
/dev/mapper/cl_-home 196G 33M 195G 1% /home
tmpfs 184M 0 184M 0% /run/user/0
ceph-fuse 54G 1.0G 53G 2% /mnt/cephfs
c720182:/ 54G 1.0G 53G 2% /mnt/cephnfs
/dev/rbd1 20G 33M 20G 1% /mnt/ceph-disk1
最新文章
- 记录一次bug解决过程:可维护性和性能优化
- 和transformjs一起摇摆
- 基于corosync+pacemaker+drbd+LNMP做web服务器的高可用集群
- [Android界面] 这样的选择器怎么实现?? 充值选择
- Linux vmstat命令--监控CPU 性能分析
- Android下 ionic view 无法登录
- Oracle和Redhat下载地址
- 我的docker 使用笔记
- mysql的sql优化案例
- Javascript个人理解
- [转]Asp.Net调用前台js调用后台代码分享
- iphone 4 safrai fixed
- 使用C#系统服务定时执行操作
- 模板语言变量,js变量,js自执行函数之前嵌套调用
- Java集合中的LinkedHashMap类
- 【MySQL 读书笔记】当我们在使用索引的时候我们在做什么
- CEPH集群操作入门--部署和运维
- Tensorflow实战系列之三:
- innerHTML的使用
- bzoj 3123 [Sdoi2013]森林(主席树+启发式合并+LCA)
热门文章
- Fedora27(CentOS7)快捷键
- MATLAB相机标定转XMl代码
- vue 优化小技巧 之 require.context()
- Java基础语法(二)
- 原生JS scroll()、scrollTo()、scrollBy()
- 2019-11-6:ubuntu 18安装tomcat 9.0
- Shell - 长 ping 脚本监控网络时延
- Selenium+Java(三)Selenium元素定位
- ctf线下赛中检测外来IP的shell脚本
- java Random类详解