Configure OpenStack network on OVH servers was indeed a toughest mission that I've ever done. OVH has MAC-IP check on gateway, public IPs which has different CIDR from the host machine, public IPs may be fragmented, which OpenStack doesn't support. I managed to solve (partial) problem by various tricks.
Constraints of OVH networks
- The OVH router only access defined MAC address, packets with strange MAC (same IP source) will be rejected. It can be tested by ping the gateway:
sudo nping --icmp --icmp-type 8 --source-mac fa:16:3e:e6:78:01 22.214.171.124 sudo nping --icmp --icmp-type 8 126.96.36.199
- RIPE IPs with different CIDR is still managed by the same router, by mapping MAC address. So VMs with RIPE IP must config gateway = gateway of host. These are the RIPE IP block that I use:
================================= IP MAC 188.8.131.52 02:00:00:c5:7a:13 184.108.40.206 02:00:00:51:23:e0 220.127.116.11 02:00:00:ee:60:a6 18.104.22.168 02:00:00:85:83:df ==================================
OpenStack uses NAT at gateway virtual router, so in this case, we need to change the MAC address of gateway virtual router interface.
OpenStack use network namespace to separate networks. So to debug/configure OpenStack network, must run this commands with the namespace. For example, run
ipconfigcommand, must use this:
sudo ip netns sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a ifconfig
- OVH ARP control: all traffic between local nodes must go through gateway (no two local nodes talking directly to each other):
sudo arp -a vss-gw-6k.fr.eu (22.214.171.124) at 00:07:b4:00:00:02 [ether] on eth0 controller (126.96.36.199) at 00:07:b4:00:00:02 [ether] on eth0
This is the output of my network config:
| br-ex: 188.8.131.52 | | | OVH | 184.108.40.206 192.168.100.1 Internet-----|gateway|-------||eth0|------------------|qg|| |qr|--(Localnet) | | 220.127.116.11 18.104.22.168 |_____| neutron router |_____| 192.168.100/24 | iptables NAT 22.214.171.124 <--> 192.168.100.40 126.96.36.199 <--> 192.168.100.41 188.8.131.52 <--> 192.168.100.41 184.108.40.206 <--- 192.168.100.0/24 <-----------OVH network-----------------> <-----------------OpenStack network----------------------->
qr: two OpenStack virtual interfaces of neutron router.
br-ex: bridge interfaces
neutron router: use an iptables NAT to translate an floating IP to an OpenStack local network.
- VM without floating IP: general NAT:
192.168.100.0/24 --> 220.127.116.11(router interface)
- VM with floating IP: two way NAT:
192.168.100.40 <--> 18.104.22.168
Prepare host networking
Run this script to bridge OVH network to OpenStack network
#! /bin/bash echo "Begin network configuration" sudo ifconfig br-ex hw ether MAC-OF-eth0 sudo ifconfig eth0 promisc up sudo ovs-vsctl add-port br-ex eth0 sudo route add default gw 22.214.171.124 br-ex sudo ifconfig br-ex 126.96.36.199 sudo ifconfig eth0 0.0.0.0 sudo service neutron-plugin-openvswitch-agent restart sudo service neutron-l3-agent restart sudo service neutron-dhcp-agent restart sudo service neutron-metadata-agent restart
Create external network
Full guide External network will stand between VMs internal network and the outside network (Internet). It must have:
- CIDR: the same with RIPE IP block (188.8.131.52 - 184.108.40.206)
- Gateway: the gateway IP of host
neutron net-create ext-net --shared --router:external=True
neutron subnet-create ext-net --name ext-subnet \ --allocation-pool start=220.127.116.11,end=18.104.22.168\ --disable-dhcp --gateway 22.214.171.124 126.96.36.199/24
Create tenant networks
Create a demo
192.168.100.1 network, can use CLI or dashboard:
- Use Dashboard:
neutron net-create demo-net neutron subnet-create demo-net --name demo-subnet 192.168.100.0/24
Create an internal router
neutron router-create demo-router
Connect external network to router
Set gateway for router is ext-net, then connect router to demo-net:
neutron router-interface-add demo-router demo-subnet neutron router-gateway-set demo-router ext-net
OVH MAC address solution
Now we come to the main trick: tackle OVH MAC-IP check on router.
Check OpenStack NAT table
sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a iptables -t nat -S
Set router gateway = default gateway of host
188.8.131.52. Add this route to Network node
sudo ip netns sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a route add 184.108.40.206/32 dev qg-1cc33b0c-7d sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a route add default gw 220.127.116.11 qg-1cc33b0c-7d
Change MAC address for router gateway port, use the MAC address of the first IP in RIPE block (18.104.22.168)
sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a ifconfig qg-1cc33b0c-7d hw ether 02:00:00:c5:7a:13
Add MAC addresses of RIPE ip block to router gateway
OVH map each IP in RIPE with a MAC address. We must add all of these MAC addresses to OVH router.
For example, if we have that RIPE:
================================================================================== IP MAC Purpose 22.214.171.124 02:00:00:c5:7a:13 Router IP, do not change 126.96.36.199 02:00:00:51:23:e0 Floating IP, can assign to VM 188.8.131.52 02:00:00:ee:60:a6 Floating IP, can assign to VM 184.108.40.206 02:00:00:85:83:df Floating IP, can assign to VM ===================================================================================
We already add the first MAC address to router gateway. Then now add the second, third, fourth MAC to gateway interface:
sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link add link qg-0103d6fa-31 address 02:00:00:51:23:e0 eth0.1 type macvlan sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link add link qg-0103d6fa-31 address 02:00:00:ee:60:a6 eth0.2 type macvlan sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link add link qg-0103d6fa-31 address 02:00:00:85:83:df eth0.3 type macvlan sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link set up eth0.1 sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link set up eth0.2 sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link set up eth0.3
Modify the security group
On dash board:
Project -> Compute -> Access & Security -> Securities Group -> Manage Rules: add ICMP, TCP port 22 rules.
Or on console:
neutron security-group-rule-create --protocol icmp \ --direction ingress --remote-ip-prefix 0.0.0.0/0 default neutron security-group-rule-create --protocol tcp \ --port-range-min 22 --port-range-max 22 \ --direction ingress --remote-ip-prefix 0.0.0.0/0 default
At this point, the VMs with floating IP assigned can be connected from a remote host. However, this solution doesn't tackle fragmented public IP block yet. As I remember correctly, someone has shared his solution to deal with multiple floating IP blocks by direct injecting to neutron floating ip database. I'll come up after testing that approach.