FortiGate-VM HA for RedHat OpenStack 10

This example shows how to set up FortiGate Clustering Protocol (FGCP) HA with two FortiGate VMs in a RedHat OpenStack 10 environment. The example includes the two FortiGate VMs connected to a private network (private01). The FortiGate-VMs protect two networks (network-l and network-r). Each network includes a CirrOS instance (cirros-l and cirros-r) for testing.

To support HA heartbeat communication, the OpenStack environment also includes a network named ha-sync configured with the subnet used by the HA heartbeat interfaces (169.254.0.0/24).

1. Setting up the networks in OpenStack
From the OpenStack environment command line, enter the following commands to create network-r and network-l and the ha-sync network.
$ source overcloudrc_tenant01 $ openstack network create network-r $ openstack subnet create subnet-r --network network-r --subnet-range 172.32.0.0/24 --dns-nameserver 208.91.112.53 $ openstack network create network-l $ openstack subnet create subnet-l --network network-l --subnet-range 172.33.0.0/24 --dns-nameserver 208.91.112.53 $ openstack network create ha-sync $ openstack subnet create subnet-ha --network ha-sync --subnet-range 169.254.0.0/24 --dns-nameserver 208.91.112.53
Add the CirrOS instances to network-r and network-l:
$ openstack server create --flavor m1.tiny --image cirros035 --security-group web --nic net-id=network-r cirros-r $ openstack server create --flavor m1.tiny --image cirros035 --security-group web --nic net-id=network-l cirros-l
2. Deploy two FortiGate-VMs
From the OpenStack command line, enter the following commands to deploy two FortiGate-VM instances (fgt-vm-1 and fgt-vm-2). These commands use the standard license files you receive when you register your FortiGate-VMs (in this example, FGVM080000103268.lic and FGVM080000109643.lic).
$ openstack server create --flavor m1.fortigate --image fgtb1486 --user-data /home/stack/openstack/cloud-init/userdata.txt --config-drive=true --file license=/home/stack/FG-licenses/FGVM080000103268.lic --security-group web --nic net-id=private01 --nic net-id=network-r --nic net-id=network-l --nic net-id=ha-sync fgt-vm-1 $ openstack server create --flavor m1.fortigate --image fgtb1486 --user-data /home/stack/openstack/cloud-init/userdata.txt --config-drive=true --file license=/home/stack/FG-licenses/FGVM080000109643.lic --security-group web --nic net-id=private01 --nic net-id=network-r --nic net-id=network-l --nic net-id=ha-sync fgt-vm-2
Here is an example `userdata.txt` file used for fgt-vm-1. The `userdata.txt` file for fgt-vm-2 would be the same except for the hostname. The `userdata.txt` file allows you to set up a FortiGate-VM with a basic default configuration customized for your environment and requirements. This example configures interfaces, adds a DNS server, and adds two firewall policies that allow any traffic to pass between the port2 and port3 interfaces. These policies make it easier to test HA failover. In addition, the MTU of the port4 interface is set to be compatible with the OpenStack 10 environment. By default, OpenStack 10 networks have an MTU of 1446. The `userdata.txt` file sets the MTU of port4 to 1400. This setting is required for the HA heartbeat interfaces to be able to communicate effectively over the ha-sync network.	#FGT VM Config File config sys global set hostname fgt-vm-1 end config system interface edit port1 set mode dhcp set allowaccess http https ssh ping next edit port2 set mode dhcp set defaultgw disable set allowaccess http https ssh ping next edit port3 set mode dhcp set defaultgw disable set allowaccess http https ssh ping next edit port4 set mtu-override enable set mtu 1400 next end config system dns set primary 208.91.112.53 end config firewall policy edit 1 set name "Allow port2 to port3" set dstintf "port2" set srcintf "port3" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" set nat enable next edit 2 set name "Allow port3 to port2" set dstintf "port3" set srcintf "port2" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" set nat enable end config system central-management set include-default-servers disable set type fortimanager set fmg 10.210.8.25 config server-list edit 1 set server-type update rating set server-address 10.210.8.25 end end
You can use the OpenStack Horizon Networks view to verify the MTU assigned to the ha-sync network.
3. Disable port security for the FortiGate-VM and CirrOS instances
Use the RedHat OpenStack Horizon Instances view to verify the IP addresses of the FortiGate-VM, the CirrOS instances, and the networks the interfaces are connected to. For example:
From the OpenStack command line, run the following bash script to disable port security on the FG-VM interfaces.
#!/bin/bash echo echo 'Disable port_security on fgt-vm-1' echo echo `source /home/stack/overcloudrc_tenant01` FGT='fgt-vm-1' IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $2}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` neutron port-update $PORTID --no-security-groups --port_security_enabled=False echo echo $IPADDR echo `openstack port show $PORTID` echo IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $3}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` `neutron port-update $PORTID --no-security-groups --port_security_enabled=False` echo echo $IPADDR echo `openstack port show $PORTID` echo IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $4}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` `neutron port-update $PORTID --no-security-groups --port_security_enabled=False` echo echo $IPADDR echo `openstack port show $PORTID` echo IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $5}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` `neutron port-update $PORTID --no-security-groups --port_security_enabled=False` echo echo $IPADDR echo `openstack port show $PORTID` echo echo 'Disable port-security on fgt-vm-2' echo FGT='fgt-vm-2' IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $2}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` neutron port-update $PORTID --no-security-groups --port_security_enabled=False echo echo $IPADDR echo `openstack port show $PORTID` echo IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $3}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` `neutron port-update $PORTID --no-security-groups --port_security_enabled=False` echo echo $IPADDR echo `openstack port show $PORTID` echo IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $4}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` `neutron port-update $PORTID --no-security-groups --port_security_enabled=False` echo echo $IPADDR echo `openstack port show $PORTID` echo IPADDR=`openstack server show $FGT \| grep addresses \| awk -F "\|" '{print $3}' \| awk -F "=" '{print $5}' \| awk -F ";" '{print $1}'` PORTID=`openstack port list \| grep $IPADDR \| awk -F "\|" '{print $2}'` `neutron port-update $PORTID --no-security-groups --port_security_enabled=False` echo echo $IPADDR echo `openstack port show $PORTID` echo
From the OpenStack command line, associate floating IPs to the two FortiGate-VMs, by entering the following commands:
openstack server add floating ip fgt-vm-1 10.210.9.10 openstack server add floating ip fgt-vm-2 10.210.9.14
4. Complete the FortiGate-VM configuration
From each FortiGate-VM instance CLI, enter the following commands to change the FortiGate-VM interfaces from DHCP to static, add IP addresses, and add a static route. The IP addresses assigned to the interfaces must be on the subnets of the networks the interfaces are connected to. The example shows the fgt-vm-1 configuration. The fgt-vm-2 configuration would be the same except for the interface IP addresses.	config system interface edit "port1" set mode static set ip 172.31.0.3 255.255.255.0 set allowaccess ping https ssh http next edit "port2" set mode static set ip 172.32.0.9 255.255.255.0 set allowaccess ping https ssh http next edit "port3" set mode static set ip 172.33.0.4 255.255.255.0 set allowaccess ping https ssh http next end config router static edit 1 set gateway 172.31.0.1 set device "port1" next end
From each FortiGate-VM instance CLI, configure both FortiGate-VMs for HA. Both FortiGate-VMs must have the same HA configuation.	config system ha set group-name "group-01" set mode a-p set password <password> set hbdev "port4" 50 set override disable set monitor "port2" end
5. CirrOS instance configuration
From each CirrOS CLI, configure each CirrOS instance with a default gateway that points at the FortiGate-VM interface connected to the same network as the CirrOS instance. Enter the following commands from each CirrOS CLI:	sudo route del default sudo ip route add default via <FG-IP-Address>
6. Testing cluster operation and failover
On the cirros-l instance console, start a continuous ping to the IP address of cirros-r. On the cirros-r instance console, start a continuous ping to the IP address of cirros-l:
On both FortiGate-VMs, use the command `diagnose sniffer packet any 'icmp' 4` to sniff ICMP packets. You should only see packets going through the primary unit.
Now shut down the primary unit. You can do this from the OpenStack Horizon Instances list.
After failover, on the new primary unit enter the command `diagnose sniffer packet any 'icmp' 4` to verify that the pings are now going through it.
7. Troubleshooting diagnose commands
On either FortiGate-VM you can use the `diagnose sys ha status` command to verify the status of the cluster.
fgt-vm # diagnose sys ha status HA information Statistics traffic.local = s:0 p:42311 b:9008646 traffic.total = s:0 p:42316 b:9009528 activity.fdb = c:0 q:0 Model=80008, Mode=2 Group=0 Debug=0 nvcluster=1, ses_pickup=0, delay=0 [Debug_Zone HA information] HA group member information: is_manage_master=1. FGVM080000109643: Master, serialno_prio=0, usr_priority=128, hostname=fgt-vm FGVM080000103268: Slave, serialno_prio=1, usr_priority=128, hostname=fgt-vm [Kernel HA information] vcluster 1, state=work, master_ip=169.254.0.1, master_id=0: FGVM080000109643: Master, ha_prio/o_ha_prio=0/0 FGVM080000103268: Slave, ha_prio/o_ha_prio=1/1
The command `get system ha status` shows similar information.
fgt-vm # get system ha status HA Health Status: OK Model: FortiGate-VM64-KVM Mode: HA A-P Group: 0 Debug: 0 Cluster Uptime: 0 days 02:04:26 Cluster state change time: 2017-09-01 03:08:19 Master selected using: <2017/09/01 03:08:19> FGVM080000109643 is selected as the master because it has the largest value of serialno. ses_pickup: disable override: disable Configuration Status: FGVM080000109643(updated 2 seconds ago): in-sync FGVM080000103268(updated 0 seconds ago): out-of-sync System Usage stats: FGVM080000109643(updated 2 seconds ago): sessions=4, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=55% FGVM080000103268(updated 0 seconds ago): sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=54% HBDEV stats: FGVM080000109643(updated 2 seconds ago): port4: physical/10000full, up, rx-bytes/packets/dropped/errors=15043566/61878/0/0, tx=158364378/146977/0/0 FGVM080000103268(updated 0 seconds ago): port4: physical/10000full, up, rx-bytes/packets/dropped/errors=29442835/61625/49/0, tx=25246662/68626/0/0 MONDEV stats: FGVM080000109643(updated 2 seconds ago): port2: physical/10000full, up, rx-bytes/packets/dropped/errors=1892/8/0/0, tx=173710/307/0/0 FGVM080000103268(updated 0 seconds ago): port2: physical/10000full, up, rx-bytes/packets/dropped/errors=174390/306/0/0, tx=2352/13/0/0 Master: fgt-vm , FGVM080000109643 Slave : fgt-vm , FGVM080000103268 number of vcluster: 1 vcluster 1: work 169.254.0.1 Master:0 FGVM080000109643 Slave :1 FGVM080000103268
The command `diagnose system ha checksum show` shows whether the configurations of the FortiGate-VMs in the cluster are synchronized. If the configurations are synchronized both sets of checksums should match.
fgt-vm # diagnose sys ha checksum show is_manage_master()=1, is_root_master()=1 debugzone global: 33 6f ee 5b 78 a5 22 84 39 ec 36 d3 1c 54 7c 78 root: 40 0d fb 04 12 41 df ad f1 64 14 03 ff ec f5 01 all: d3 2f 6f bb a6 e7 77 db 27 75 81 b2 94 f3 fd 68 checksum global: 33 6f ee 5b 78 a5 22 84 39 ec 36 d3 1c 54 7c 78 root: 40 0d fb 04 12 41 df ad f1 64 14 03 ff ec f5 01 all: d3 2f 6f bb a6 e7 77 db 27 75 81 b2 94 f3 fd 68
If the checksums do not match, you can use the `diagnose sys ha checksum show` and `diagnose sys ha checksum show global` commands to show more detailed checksum results, the following example shows the first few lines of output of the `diagnose sys ha checksum show global` command:
diagnose sys ha checksum show global system.global: 2c79958c132639dfe61ab782a2f213ec system.accprofile: 7d79452c78377be2616149264a18fd5c system.vdom-link: 00000000000000000000000000000000 wireless-controller.inter-controller: 00000000000000000000000000000000 wireless-controller.global: 00000000000000000000000000000000 wireless-controller.vap: 00000000000000000000000000000000 system.switch-interface: 00000000000000000000000000000000 system.interface: 8690699bc33c7c15b20e017876cf1e37 ...
If the configurations are synchronized all of the checksums displayed using these commands from both of the FortiGate-VMs should match. If they do not you can use the output to see what parts of the configuration are not synchronized.