Heartbeat+DRBD实现NFS的高可用

DRBD(全称为Distributed Replicated Block Device,简称drbd)分布式块设备复制,就是在不同节点上两个相同大小的设备块级别之间的数据同步镜像。DRBD是由内核模块和相关脚本而构成,用以构建高可用性的集群。主节点上的DRBD提升为Primary并负责接收写入数据,当数据到达DRBD模块时,一份继续往下走写入到本地磁盘实现数据的持久化,同时并将接收到的 要写入的数据发送一分到本地的DRBD设备上通过tcp传到另外一台主机的DRBD设备上(Secondary node),另一台主机上的对应的DRBD设备再将接收到的数据存入到自己的磁盘当中。这里与mysql的基于通过二进制日志完成数据的复制的确很相似, 但是也有一些不同之处。比如:mysql的从节点不能写但是可以读,但是DRBD的从节点是不能读、不能挂载。 因此,DRBD对同一设备块每次只允许对主节点进行读、写操作,从节点不能写也不能读。

以上说明出自http://www.68idc.cn/help/jiabenmake/qita/20150922558664.html

下面是我做的实验。

VMWare  Ubuntu12.04

Aaron     172.16.18.181

THEO     172.16.18.227

在两台虚拟机/etc/hosts下添加
172.16.18.227     THEO
172.16.18.181     Aaron
并分别安装用到的软件
apt-get -y install heartbeat
apt-get -y install drbd8-utils
apt-get -y install nfs-kernel-server

 

在两台虚拟机上分别添加一块2G的硬盘

root@THEO:~# fdisk -l


Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a683b


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    40894463    20446208   83  Linux
/dev/sda2        40896510    41940991      522241    5  Extended
/dev/sda3        40894464    40896509        1023   83  Linux
/dev/sda5        40896512    41940991      522240   82  Linux swap / Solaris


Partition table entries are not in disk order


Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdb doesn't contain a valid partition table


root@THEO:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xc4c7e12d.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.


Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)


Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-4194303, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4194303, default 4194303):
Using default value 4194303


Command (m for help): w
The partition table has been altered!


Calling ioctl() to re-read partition table.
Syncing disks.

 

两台虚拟机上的DRBD配置文件保持一致
root@THEO:/etc/drbd.d# more nfs.res
resource nfs {
     protocol C;
     startup {
          wfc-timeout 15;
          degr-wfc-timeout 60;
     }
     net {
          after-sb-0pri discard-zero-changes;
          after-sb-1pri discard-secondary;
          after-sb-2pri disconnect;
     }
     syncer {
          rate 20M;
     }
     on Aaron {
          device /dev/drbd0;
          disk /dev/sdb1;
          address 172.16.18.181:7898;
          meta-disk internal;
     }
     on THEO {
          device /dev/drbd0;
          disk /dev/sdb1;
          address 172.16.18.227:7898;
          meta-disk internal;
     }
}

 

在Aaron上创建DRBD存储设备
drbdadm create-md nfs
/etc/init.d/drbd start
/etc/init.d/drbd status
drbdsetup /dev/drbd0 primary -o
mkfs.ext3 /dev/drbd0
mkdir /usr/local/datashare
mount /dev/drbd0 /usr/local/datashare

 

停止Aaron上的drbd服务,然后在THEO上创建DRBD存储设备
drbdadm create-md nfs
/etc/init.d/drbd start
/etc/init.d/drbd status
drbdsetup /dev/drbd0 primary -o
mkfs.ext3 /dev/drbd0
mkdir /usr/local/datashare
mount /dev/drbd0 /usr/local/datashare

 

配置Heartbeat
root@Aaron:/etc/ha.d# more /etc/ha.d/ha.cf |grep -v ^#
logfile     /var/log/ha-log
keepalive 2
deadtime 10
warntime 5
initdead 60
udpport     694
bcast     eth0          # Linux
ucast eth0 172.16.18.181
auto_failback on
node     Aaron
node     THEO
ping 172.16.18.2
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster

root@Aaron:/etc/ha.d# more haresources |grep -v ^#
Aaron IPaddr::172.16.18.200/24/eth0 drbddisk::nfs Filesystem::/dev/drbd0::/usr/local/datashare::ext3 killnfsd

root@Aaron:/etc/ha.d# more authkeys |grep -v ^#
auth 1
1 sha1 passwdanyway

root@THEO:/etc/ha.d# more /etc/ha.d/ha.cf |grep -v ^#
logfile     /var/log/ha-log
keepalive 2
deadtime 10
warntime 5
initdead 60
udpport     694
bcast     eth0          # Linux
ucast eth0 172.16.18.227
auto_failback on
node     Aaron
node     THEO
ping 172.16.18.2
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster

root@THEO:/etc/ha.d# more haresources |grep -v ^#
Aaron IPaddr::172.16.18.200/24/eth0 drbddisk::nfs Filesystem::/dev/drbd0::/usr/local/datashare::ext3 killnfsd


root@THEO:/etc/ha.d# more authkeys |grep -v ^#
auth 1
1 sha1 passwdanyway

 

DRBD服务

/etc/init.d/drbd start
/etc/init.d/drbd stop
/etc/init.d/drbd restart
/etc/init.d/drbd status

 

一些命令
drbdadm secondary nfs        #强制为从
drbdadm primary nfs          #强制为主
cat /proc/drbd               #查看drdb状态

 

以下手动恢复Split-Brain状况: i. 在secondary上:
drbdadm secondary nfs
drbdadm disconnect all
drbdadmin -- --discard-my-data connect nfs

 

ii.在primary上:
drbdadm disconnect all
drbdadm connect nfs

 

参考资料: