一、服务器规划

主机名 主机IP 磁盘 角色
node3 public-ip:172.18.112.20
cluster-ip: 172.18.112.20
vdb ceph-deploy,monitor,mgr,osd
node4 public-ip:172.18.112.19
cluster-ip: 172.18.112.19
vdb monitor,mgr,osd
node5 public-ip:172.18.112.18
cluster-ip: 172.18.112.18
vdb monitor,mgr,osd

二、设置主机名

主机名设置,三台主机分别执行属于自己的命令
node3

1
2
[root@localhost ~]# hostnamectl set-hostname nod3
[root@localhost ~]# hostname node3

node4

1
2
3
[root@localhost ~]# hostnamectl set-hostname node4
[root@localhost ~]# hostname node4

node5

1
2
3
[root@localhost ~]# hostnamectl set-hostname node5
[root@localhost ~]# hostname node5

执行完毕后要想看到效果,需要关闭当前命令行窗口,重新打开即可看到设置效果

三、设置hosts文件

在3台机器上都执行下面命令,添加映射

1
2
3
echo "172.18.112.20 node3 " >> /etc/hosts
echo "172.18.112.19 node4 " >> /etc/hosts
echo "172.18.112.18 node5 " >> /etc/hosts

四、创建用户并设置免密登录

创建用户(三台机器上都运行)

1
2
3
4
5
useradd -d /home/admin -m admin
echo "123456" | passwd admin --stdin
#sudo权限
echo "admin ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/admin
sudo chmod 0440 /etc/sudoers.d/admin

设置免密登录 (只在node3上执行)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@node3 ~]# su - admin
[admin@node3 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/admin/.ssh/id_rsa):
Created directory '/home/admin/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/admin/.ssh/id_rsa.
Your public key has been saved in /home/admin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:qfWhuboKeoHQOOMLOIB5tjK1RPjgw/Csl4r6A1FiJYA admin@admin.ops5.bbdops.com
The key's randomart image is:
+---[RSA 2048]----+
|+o.. |
|E.+ |
|*% |
|X+X . |
|=@.+ S . |
|X.* o + . |
|oBo. . o . |
|ooo. . |
|+o....oo. |
+----[SHA256]-----+
[admin@node3 ~]$ ssh-copy-id admin@node3
[admin@node3 ~]$ ssh-copy-id admin@node4
[admin@node3 ~]$ ssh-copy-id admin@node5

注意: 没有ssh-copy-id 这个命令可以手动把公钥传到对应的机器上去

1
cat ~/.ssh/id_*.pub | ssh  admin@host3 'cat >> .ssh/authorized_keys'

五、配置时间同步

三台都执行

1
2
3
4
5
6
7
8
9
10
11
[root@node3 ~]$ timedatectl #查看本地时间

[root@node3 ~]$ timedatectl set-timezone Asia/Shanghai #改为亚洲上海时间

[root@node3 ~]$ yum install -y chrony #同步工具

[root@node3 ~]$ chronyc -n sources -v #同步列表

[root@node3 ~]$ chronyc tracking #同步服务状态

[root@node3 ~]$ timedatectl status #查看本地时间

六、安装ceph-deploy并安装ceph软件包

配置ceph清华源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat > /etc/yum.repos.d/ceph.repo<<'EOF'
[Ceph]
name=Ceph packages for $basearch
baseurl=https://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1
[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1
[ceph-source]
name=Ceph source packages
baseurl=https://mirror.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirror.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1
EOF

安装ceph-deploy

1
[admin@node3 ~]# sudo yum install ceph-deploy

初始化mon点

ceph需要epel源的包,所以安装的节点都需要yum install epel-release

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[admin@node3 ~]$ mkdir my-cluster
[admin@node3 ~]$ cd my-cluster
# new
[admin@node3 my-cluster]$ ceph-deploy new node3 node4 node5
Traceback (most recent call last):
File "/bin/ceph-deploy", line 18, in <module>
from ceph_deploy.cli import main
File "/usr/lib/python2.7/site-packages/ceph_deploy/cli.py", line 1, in <module>
import pkg_resources
ImportError: No module named pkg_resources
#以上出现报错,是因为没有pip,安装pip
[admin@node3 my-cluster]$ sudo yum install epel-release
[admin@node3 my-cluster]$ sudo yum install python-pip
#重新初始化
[admin@node3 my-cluster]$ ceph-deploy new node3 node4 node5
[admin@node3 my-cluster]$ ls
ceph.conf ceph-deploy-ceph.log ceph.mon.keyring
[admin@node3 my-cluster]$ cat ceph.conf
[global]
fsid = 3a2a06c7-124f-4703-b798-88eb2950361e
mon_initial_members = node3, node4, node5
mon_host = 172.18.112.20,172.18.112.19,172.18.112.18
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

修改ceph.conf,添加如下配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
public network = 172.18.112.0/24
cluster network = 172.18.112.0/24
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 128
osd pool default pgp num = 128
osd pool default crush rule = 0
osd crush chooseleaf type = 1
max open files = 131072
ms bind ipv6 = false
[mon]
mon clock drift allowed = 10
mon clock drift warn backoff = 30
mon osd full ratio = .95
mon osd nearfull ratio = .85
mon osd down out interval = 600
mon osd report timeout = 300
mon allow pool delete = true
[osd]
osd recovery max active = 3
osd max backfills = 5
osd max scrubs = 2
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=1024
osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog
filestore max sync interval = 5
osd op threads = 2

安装Ceph软件到指定节点

1
[admin@node3 my-cluster]$ ceph-deploy install --no-adjust-repos node3 node4 node5

–no-adjust-repos是直接使用本地源,不生成官方源.

部署初始的monitors,并获得keys

1
[admin@nod3 my-cluster]$ ceph-deploy mon create-initial

做完这一步,在当前目录下就会看到有如下的keyrings:

1
2
3
[admin@node3 my-cluster]$ ls
ceph.bootstrap-mds.keyring ceph.bootstrap-osd.keyring ceph.client.admin.keyring ceph-deploy-ceph.log
ceph.bootstrap-mgr.keyring ceph.bootstrap-rgw.keyring ceph.conf ceph.mon.keyring

将配置文件和密钥复制到集群各节点

配置文件就是生成的ceph.conf,而密钥是ceph.client.admin.keyring,当使用ceph客户端连接至ceph集群时需要使用的密默认密钥,这里我们所有节点都要复制,命令如下。

1
[admin@node3 my-cluster]$ ceph-deploy admin node3 node4 node5

七、部署ceph-mgr

1
2
#在L版本的`Ceph`中新增了`manager daemon`,如下命令部署一个`Manager`守护进程
[admin@node3 my-cluster]$ ceph-deploy mgr create node3

八、创建osd

1
2
3
4
#用法:ceph-deploy osd create –data {device} {ceph-node}
ceph-deploy osd create --data /dev/vdb node3
ceph-deploy osd create --data /dev/vdb node4
ceph-deploy osd create --data /dev/vdb node5

检查osd状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[admin@node3 my-cluster]$ sudo ceph health
HEALTH_OK

[admin@node3 my-cluster]$ sudo ceph -s
cluster:
id: 3a2a06c7-124f-4703-b798-88eb2950361e
health: HEALTH_OK

services:
mon: 3 daemons, quorum node5,node4,node3
mgr: node3(active)
osd: 3 osds: 3 up, 3 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 MiB
usage: 3.2 GiB used, 597 GiB / 600 GiB avail
pgs:

默认情况下ceph.client.admin.keyring文件的权限为600,属主和属组为root,如果在集群内节点使用cephadmin用户直接直接ceph命令,将会提示无法找到/etc/ceph/ceph.client.admin.keyring文件,因为权限不足。

如果使用sudo ceph不存在此问题,为方便直接使用ceph命令,可将权限设置为644。在集群节点上面node1 admin用户下执行下面命令。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[admin@node3 my-cluster]$ ceph -s
2021-12-28 07:59:36.062 7f52d08e0700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-12-28 07:59:36.062 7f52d08e0700 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication
[errno 2] error connecting to the cluster
[admin@node3 my-cluster]$ sudo chmod 644 /etc/ceph/ceph.client.admin.keyring


[admin@node3 my-cluster]$ ceph -s
cluster:
id: 3a2a06c7-124f-4703-b798-88eb2950361e
health: HEALTH_OK

services:
mon: 3 daemons, quorum node5,node4,node3
mgr: node3(active)
osd: 3 osds: 3 up, 3 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 MiB
usage: 3.2 GiB used, 597 GiB / 600 GiB avail
pgs:

查看osds

1
2
3
4
5
6
7
8
9
[admin@node3 my-cluster]$ sudo ceph osd tree 
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.58589 root default
-3 0.19530 host node3
3 hdd 0.19530 osd.3 up 1.00000 1.00000
-5 0.19530 host node4
4 hdd 0.19530 osd.4 up 1.00000 1.00000
-7 0.19530 host node5
5 hdd 0.19530 osd.5 up 1.00000 1.00000

九、开启MGR监控模块

方式一:命令操作

1
ceph mgr module enable dashboard

如果以上操作报错如下:

1
Error ENOENT: all mgr daemons do not support module 'dashboard', pass --force to force enablement

则因为没有安装ceph-mgr-dashboard,在mgr的节点上安装。

1
2
yum install ceph-mgr-dashboard

方式二:配置文件

1
2
3
4
5
6
7
8
9
# 编辑ceph.conf文件
vi ceph.conf
[mon]
mgr initial modules = dashboard
#推送配置
[admin@node3 my-cluster]$ ceph-deploy --overwrite-conf config push node3 node4 node5
#重启mgr
sudo systemctl restart ceph-mgr@node3

web登录配置
默认情况下,仪表板的所有HTTP连接均使用SSL/TLS进行保护。

1
2
3
4
5
6
7
#要快速启动并运行仪表板,可以使用以下内置命令生成并安装自签名证书:
[root@node3 my-cluster]# ceph dashboard create-self-signed-cert
Self-signed certificate created

#创建具有管理员角色的用户:
[root@node3 my-cluster]# ceph dashboard set-login-credentials admin admin
Username and password updated

#查看ceph-mgr服务:

1
2
3
4
[root@node3 my-cluster]# ceph mgr services
{
"dashboard": "https://node3:8443/"
}

以上配置完成后,浏览器输入 https://node3:8443 输入用户名admin,密码admin登录即可查看

要本地hosts解析

ceph