langyal’s blog

snippets related to my work and interest

Openstack in Openstack

GOAL of this post: to run Openstack on top of Openstack.

use cases:

  • learning
  • testing
  • coding

high level considerations

  • I have access to an openstack environment
    • I have a tenant/project with enough resources to run ~20 instances, each 1 vcpu, ~8GB RAM, ~40GB disk, ~100GB volume
    • I have a standard user account
  • I have just one public FIP
    • if I had two, I could have a snat router (FIP1) plus a jumphost (FIP2)
    • since I have just one, I have to proxy everything on the jumphost (FIP)
  • I need L2 connectivity between the vm-in-vm instances
    • this is not possible in a typical cloud
    • so, I need a trick: a VXLAN overlay on top of the existing openstack network

high level solution

  • I set up a VXLAN overlay on top of the existing openstack network
  • I will run my underlay VMs as standard openstack instances (named uc-XXXX)
  • I will run my overlay VMs as VMs in openstack instances (named oc-XXXX)
    • maas, juju
    • openstack building units
      • barematel = vm-in-vm
      • lxd = lxd-in-vm-in-vm
  • networks
    • 10.22.0.0/16:
      • a standard openstack network in your “underlay openstack”
      • used by the undercloud openstack instances: r0,uc-XXXX
    • 10.33.0.0/16:
      • overlay network with 50 bytes less MTU
      • used by the overcloud instances (vm-in-vm): oc-XXXX
    • 10.44.0.0/16:
      • over-overcloud network: created in the over-openstack
      • used by the vm-in-vm-in-vm instances as external network
    • 10.55.0.0/16
      • over-overcloud network: created in the over-openstack
      • used by the vm-in-vm-in-vm instances as internal network
  • interfaces
    • r0
      • ens3: ip:10.22.0.3/16
      • vxlan10 on the bridge br-vxlan10:
        • gw/snat for 10.33.0.0/16: ip:10.33.0.1
        • gw/snat for 10.44.0.0/16: ip:10.44.0.1
    • uc-XXXX
      • ens3: 10.22.X.X/16
      • vxlan10 on the bridge br-vxlan10: no ip assigned
    • oc-XXXX
      • eth0: 10.33.X.X/16
      • eth1: no ip assigned
  • I will run my openstack-in-openstack instances as VMs in an overlay VM with virt-type=qemu
    • yes, it is a vm-in-vm-in-vm
    • yes, it works :)

undercloud

  • overview
    • create a VM with following services:
      • one public IP (referred to as FIP=Floating IP)
      • jumphost
      • http/https proxy
      • name resolver
      • ntp server
      • admin client for ansible, maas, juju, openstack
    • prepare underlay VMs
  • step-by-step instructions: 01-undercloud-deploy.md
  • undercloud design: undercloud

overcloud: prepare

  • overview
    • maas
      • create maas vm-in-vm with configdrive
        • since we don’t have dhcp/metadata service on this network
        • dhcp/pxe boot will be provided by maas itself once it’s running
      • maas configuration
    • feeding maas with “empty” instances (vm-in-vm)
  • step-by-step instructions: 02-overcloud-prepare.md
  • overcloud design: overcloud

overcloud: deploy

  • overview

    • juju client
    • juju controller
    • neutron-gateway preparation
    • deploy the bundle with juju

  • step-by-step instructions: 03-overcloud-deploy.md

  • maas after the deployment: maas

overcloud: use

  • overview

    • how to use the openstack-in-openstack
    • how to launch and access an instance which is a vm-in-vm-in-vm

  • step-by-step instructions: 04-overcloud-use.md

references

todo

  • use bigger flavors
    • the over-hypervisor has only 3GB RAM
    • with 8GB RAM I can run only two converged over-hypervisor per uc-XXXX node
  • maas
    • deploy maas in bionic
    • deploy maas in HA
  • juju
    • deploy the juju controller in bionic
    • deploy the juju controller in HA
  • neutron-gateway
    • deploy 2 more instances on uc-node12 and uc-node13
    • add them to maas
  • more automation
    • r0
    • maas
  • more separation
    • we could have 10.33/16 and 10.44/16 on separate vxlan
    • the script overlay-config.sh is prepared for it