A Cloudy Blog

Configure OpenStack network on OVH

Mon, 07 Jul 2014 16:00:00 +0000

Configure OpenStack network on OVH servers was indeed a toughest mission that I've ever done. OVH has MAC-IP check on gateway, public IPs which has different CIDR from the host machine, public IPs may be fragmented, which OpenStack doesn't support. I managed to solve (partial) problem by various tricks.

Constraints of OVH networks

The OVH router only access defined MAC address, packets with strange MAC (same IP source) will be rejected. It can be tested by ping the gateway:

sudo nping --icmp --icmp-type 8 --source-mac fa:16:3e:e6:78:01 176.31.105.254
sudo nping --icmp --icmp-type 8 176.31.105.254

RIPE IPs with different CIDR is still managed by the same router, by mapping MAC address. So VMs with RIPE IP must config gateway = gateway of host. These are the RIPE IP block that I use:

=================================
IP              MAC
46.105.252.216  02:00:00:c5:7a:13
46.105.252.217  02:00:00:51:23:e0
46.105.252.218  02:00:00:ee:60:a6
46.105.252.219  02:00:00:85:83:df
==================================

OpenStack uses NAT at gateway virtual router, so in this case, we need to change the MAC address of gateway virtual router interface.
OpenStack use network namespace to separate networks. So to debug/configure OpenStack network, must run this commands with the namespace. For example, run ipconfig command, must use this:

sudo ip netns
sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a ifconfig

instead this:

sudo ifconfig

OVH ARP control: all traffic between local nodes must go through gateway (no two local nodes talking directly to each other):

sudo arp -a
vss-gw-6k.fr.eu (176.31.105.254) at 00:07:b4:00:00:02 [ether] on eth0
controller (176.31.105.64) at 00:07:b4:00:00:02 [ether] on eth0

Overview

This is the output of my network config:

                             |    br-ex: 176.31.255.64    |
                             |                            | 
                OVH          |                    46.105.252.216               192.168.100.1
Internet-----|gateway|-------||eth0|------------------|qg||                         |qr|--(Localnet)
                                                        |                            |
          176.31.255.254     176.31.255.64              |_____| neutron router |_____| 192.168.100/24                                                                       |      
                                                                   iptables NAT                                                                              46.105.252.217 <--> 192.168.100.40
                                                        46.105.252.218 <--> 192.168.100.41
                                                        46.105.252.219 <--> 192.168.100.41
                                                        46.105.252.216 <--- 192.168.100.0/24 
<-----------OVH network-----------------> <-----------------OpenStack network----------------------->

qg, qr: two OpenStack virtual interfaces of neutron router.
br-ex: bridge interfaces eth0 and qg
neutron router: use an iptables NAT to translate an floating IP to an OpenStack local network.
VM without floating IP: general NAT: 192.168.100.0/24 --> 46.105.252.216 (router interface)
VM with floating IP: two way NAT: 192.168.100.40 <--> 46.105.252.217

Prepare host networking

Run this script to bridge OVH network to OpenStack network

#! /bin/bash
echo "Begin network configuration"
sudo ifconfig br-ex hw ether MAC-OF-eth0
sudo ifconfig eth0 promisc up
sudo ovs-vsctl add-port br-ex eth0

sudo route add default gw 176.31.105.254 br-ex
sudo ifconfig br-ex 176.31.105.64
sudo ifconfig eth0 0.0.0.0

sudo service neutron-plugin-openvswitch-agent restart
sudo service neutron-l3-agent restart
sudo service neutron-dhcp-agent restart
sudo service neutron-metadata-agent restart

Create external network

Full guide External network will stand between VMs internal network and the outside network (Internet). It must have:

CIDR: the same with RIPE IP block (46.105.252.216 - 46.105.252.219)
Gateway: the gateway IP of host

neutron net-create ext-net --shared --router:external=True

neutron subnet-create ext-net --name ext-subnet \
  --allocation-pool start=46.105.252.216,end=46.105.252.219\
    --disable-dhcp --gateway 176.31.105.254 46.105.252.216/24

Create tenant networks

Create a demo 192.168.100.1 network, can use CLI or dashboard:

Use Dashboard: Project->Network->Networks->Create Network
Console:

neutron net-create demo-net
neutron subnet-create demo-net --name demo-subnet 192.168.100.0/24

Create an internal router

Dashboard: Project->Network->Routers->Create Router
Console:

neutron router-create demo-router

Connect external network to router

Set gateway for router is ext-net, then connect router to demo-net:

Dashboard: Project->Network->Routers
Console:

neutron router-interface-add demo-router demo-subnet
neutron router-gateway-set demo-router ext-net

OVH MAC address solution

Now we come to the main trick: tackle OVH MAC-IP check on router.

Check OpenStack NAT table

sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a iptables -t nat -S

Set router gateway = default gateway of host `176.31.105.254`. Add this route to Network node

sudo ip netns
sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a route add 176.31.105.254/32 dev qg-1cc33b0c-7d
sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a route add default gw 176.31.105.254 qg-1cc33b0c-7d

Change MAC address for router gateway port, use the MAC address of the first IP in RIPE block (46.105.252.216)

sudo ip netns exec qrouter-d1ceaf19-4cf0-4334-9f96-7601c70cbf4a ifconfig qg-1cc33b0c-7d hw ether 02:00:00:c5:7a:13

Add MAC addresses of RIPE ip block to router gateway

OVH map each IP in RIPE with a MAC address. We must add all of these MAC addresses to OVH router.

For example, if we have that RIPE:

==================================================================================
IP              MAC                     Purpose
46.105.252.216  02:00:00:c5:7a:13       Router IP, do not change
46.105.252.217  02:00:00:51:23:e0       Floating IP, can assign to VM
46.105.252.218  02:00:00:ee:60:a6       Floating IP, can assign to VM
46.105.252.219  02:00:00:85:83:df       Floating IP, can assign to VM
===================================================================================

We already add the first MAC address to router gateway. Then now add the second, third, fourth MAC to gateway interface:

sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link add link qg-0103d6fa-31 address 02:00:00:51:23:e0 eth0.1 type macvlan
sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link add link qg-0103d6fa-31 address 02:00:00:ee:60:a6 eth0.2 type macvlan
sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link add link qg-0103d6fa-31 address 02:00:00:85:83:df eth0.3 type macvlan
sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link set up eth0.1
sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link set up eth0.2
sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf ip link set up eth0.3

Modify the security group

On dash board: Project -> Compute -> Access & Security -> Securities Group -> Manage Rules: add ICMP, TCP port 22 rules.

Or on console:

neutron security-group-rule-create --protocol icmp \
  --direction ingress --remote-ip-prefix 0.0.0.0/0 default

  neutron security-group-rule-create --protocol tcp \
    --port-range-min 22 --port-range-max 22 \
      --direction ingress --remote-ip-prefix 0.0.0.0/0 default

Conclusion

At this point, the VMs with floating IP assigned can be connected from a remote host. However, this solution doesn't tackle fragmented public IP block yet. As I remember correctly, someone has shared his solution to deal with multiple floating IP blocks by direct injecting to neutron floating ip database. I'll come up after testing that approach.

Introduction to OpenCloud

Tue, 22 Apr 2014 16:00:00 +0000

The Cloud is one of the most popular term on Internet nowadays. Cloud computing is widely adopted by all scaled enterprises. By wikipedia definition, cloud computing is computing that involves a large number of computers connected through a communication network such as the Internet, similar to utility computing. In science, cloud computing is a synonym for distributed computing over a network, and means the ability to run a program or application on many connected computers at the same time.

Infrastructure as a Service (IaaS) is one of the three fundamental services of cloud computing. It aims for providing a hardware infrastructure, including the virtual servers, network connections, IP address, storage hardwares, etc. or a pool of resources. This resource pool can be scale easily later.

One well-known example of IaaS provider is Amazon Web Service (AWS). They provide users scalable clusters with unlimited VMs. Users don't care about networking or hardwares below. Users can add more storage space, CPU, RAM for their VMs anytime.

AWS is considered public cloud. Amazon controls the hardware, and all users share that resource pool. All users' data is stored on Amazon's hardware property. That create the fear of vendor lock-in - a day when the IaaS refuse to serve and claim all data on their servers. That gives birth for the Private Cloud trend - the cloud that runs on enterprise's own hardware. On the market, there are plenty of Open Source and commercial product that help building private cloud:

OpenStack
CloudStack
Eucalyptus

They are all open source. Among them, OpenStack is having the momentum. They have largest community, as well as supported by many companies: HP, IBM, Rackspace, NASA, Mirantis, etc.

http://en.wikipedia.org/wiki/Cloud-computing

What is DevOps.

Tue, 22 Apr 2014 14:00:00 +0000

I'm not an expert in DevOps myself. This blog is merely a digest note from my reading and very little experience. I'm intending to update it when I gain more experience or from your help.

What is DevOps

DevOps stands for Development and Operations. First of all, it's an "another" software development methodology. Of course, as other siblings, this methodology is created to solve some problem in software world.

The problem

Simply speaking, development guys and operations guys are enemy of each other. Development means to change and evolve. While operations need stability. Some statistic numbers: 80% downtime is due to changes. 47% of total time dedicated to deployment.

But of course we need development. The problem is the misunderstanding between Dev and Ops, that we call The wall of Confusion. To add fuel into the fire, many global companies have Dev team in Vietnam and Ops team in Europe. Geometry distance will enlarge the wall.

The worst day is release/deployment day, when changes are pushed from Dev team to Ops team. The most typical problem is the differences in environment between Dev and Ops. The Ops team has to hack the product, modify all configuration files. And when they give up, changes will be rolled back in Ops, bugs are reported to Devs, and Dev guys will say f**k

The solution

DevOps focuses on communication. Cooperation between Dev and Ops must start before and continues long after release/deployment.

We should also considering the voices again DevOps. I read an article of Jeff Knupp, and I think his point of view is interesting (not that I agree). He complained force a developer to handle tasks of QA, sysadmin, analyst. That's the birth of "full-stack" developer, and some traditional dev who only love programming will feel annoyed (not in my case definitely).

Not so convinced?

Which companies used DevOps

All of software companies can use it, of course. But the one who need it the most is the companies with very frequent releases. For example, Flickr with 10 deployments per day. Facebook with 1 deployment each day. Amazon with a deployment per 11 seconds.

Ok I'll use DevOps, but how to use?

As mentioned above, DevOps is a methodology. That means you need tools, training, changes in approach, changes in culture, etc. to adopt it. The following is just some tips based on my experience to help.

Chef: with chef, you write the code to configure the infrastructure (infrastructure as a code). The good point is you don't have to worry about the different in env between Dev and Ops. You can check this slide for more info.
Docker: provide a unique sandbox environment for apps. It's quite popular now, and is a competitor of Chef.
The culture: train dev with QA, sysadmin roles. Train sysadmin how to program. Build a team with both of them.
Deploying: deploy frequently. A deploy with smaller code changes is better than an upgrade with hundred files changes.

This slides provides plenty good examples.

Now it's your turn to help. Please sharing your own experience using DevOps.

Which OpenCloud I choose? OpenStack, CloudStack or VMware

Tue, 15 Apr 2014 15:52:47 +0000

In recent years, we can observer a rapid advance of cloud IaaS technology. In the past, only VMware made its name and widely used in DC management. Then the rise of AWS make a huge impact on the industry. To fight again the Godzilla EC2, and with the needs of private cloud, latecomers enterprises and communities join force to create different OpenSource IaaS platform:

Eucalyptus: a second broadest community and an official partner with AWS.
CloudStack: a most completed product, used to be an closed software belong to Citrix. They decide to contribute source code to Apache to join the race. Small but well-tested.
OpenNebula: with smallest community.
OpenStack: with broadest community, back by giants, and illimited pocket of money.

All of them joined a fierce stack war. I can't say which of them will win, but we will focus on the leading group: CloudStack - a number one in term of industry deploying, OpenStack - a number one in term of community and supports, and VMware - an ancient giant.

OpenCloud vs VMware

The most traditional technology for DC virtualization is VMware vCenter. Why I called it traditional. Because this tool has been developed for a long time, and was adopted by many legacy systems. However, we hardly called it Cloud technology. I have a honor to consult Mr. Thierry Carrez - OpenStack release manager. He gave me an answer that I found the most correct:

Traditional workloads run on servers that are like pets. They are unique, you care for them, heal them when they are hurt, have HA in place to make sure they survive etc.

Cloud servers are like cattle. They are generic, are set up automatically using automation, and when they exhibit signs of failure you shoot them.

If your goal is to virtualize traditional workloads, you may find VMWare offering more attractive, with nice HA / live migration features to ensure the survival of your hand-crafted servers.
If your goal is more to heavily use automation, develops style deployments and Amazon AWS like services, then OpenStack is the way to go.

Pet and cattle analogy

Why not CloudStack?

CloudStack is the top competitor of OpenStack in OpenCloud market now.

CloudStack is obvious much product oriented. I think OpenStack and CloudStack is similar to Windows vs Linux kernel.

Although CloudStack is open-source, but recently report, Citrix contributes more than 50% to the project.
While OpenStack has Rackspace, IBM, HP, Redhat, Mirantis are top contributors, and dozen of companies contributes tiny parts. In addition, each platinum contributer has their own product-tested version based on core OpenStack.

OpenStack

The concept is similar to Linux kernel and Linux distro: OpenStack is a framework (Linux kernel). Some of the companies in that ecosystem package the solution so that it's ready to use (Linux distro):

Rackspace Private Cloud (Rackspace)
Linux distributions like Red Hat or Ubuntu (Canonical) have offerings OpenStack package.
ENovance (France/Canada)
Cloudscaling (US)
Mirantis (US/Russia)
IBM (commercial version)

Rackspace Private Cloud

Rackspace Private Cloud is an OpenStack configuration that has been developed, tested, and packaged by Rackspace OpenStack specialists. Offering that is easy to install, with a configuration that comes from our expertise in developing OpenStack clouds for our customers.

Strength

Easy to install.
Well documented.
One deployment design (designed HA, networking, etc).
Based on latest release version of OpenStack (Havana)
Using Chef cookbook.
Central forum for discussion. Fast (free) support.
Rackspace is the company contributed the most for OpenStack recently.

Weakness

Not flexible (must follow Rackspace deployment design).
Use latest OpenStack release version, not development version.
Lack of OpenStack new features (incubating projects). Not sure can we add them manually.
Not sure OpenStack third parties extension can work.

My Rackspace chef environment

Tue, 15 Apr 2014 13:52:47 +0000

This is a note of how I change Rackspace Private Cloud (RPC) chef's environment before bootstrap own nodes.

{
   "name": "MDI-private-cloud",
   "description": "Mobile Device - Rackspace Private Cloud v4.2.2",
   "cookbook_versions": {
   },
   "json_class": "Chef::Environment",
   "chef_type": "environment",
   "default_attributes": {
   },
   "override_attributes": {
     "custom_template_banner": "Mobile Devices Ingenierie Cloud",
     "developer_mode": false,
     "enable_testing_repos": false,
         "nova": {
            "network": {
                "provider": "neutron"
            }
          },
         "neutron": {
            "ovs": {
            "external_bridge": "",
                "provider_networks": [
                  {
                     "label": "ph-eth0",
                     "bridge": "br-eth0",
                     "vlans": "1:1"
                   }
             ],
             "network_type": "gre"
         }
      },
      "mysql": {
         "allow_remote_root": true,
         "root_network_acl": "%"
      },
      "osops_networks": {
         "nova": "192.168.1.0/24",
         "public": "192.168.1.0/24",
         "management": "192.168.1.0/24"
      }
   }
}