Scope: this walks through how this lab’s OpenStack cloud was actually built, end to end — physical network → MikroTik BGP → OpenStack-Ansible install → internal HAProxy. Real values from this deployment (VLAN IDs, AS numbers, CIDRs, commands) are included throughout; credentials, private keys and tokens are intentionally omitted — see the referenced files directly in the repo for those, and rotate anything you find that’s ever touched git history.
1. Architecture at a glance
One physical trunk carries every VLAN into a single MikroTik RB5009, which does two jobs: it’s the L3 gateway for each VLAN, and it’s a BGP router peering with every cluster’s CNI (Calico) so pod/service CIDRs get real routes instead of NAT tricks. OpenStack itself is a 3-node OpenStack-Ansible (OSA) deployment where each of the 3 physical hosts plays every infrastructure role at once (no dedicated controller/compute split).
2. The physical network: one trunk, many VLANs
Every VLAN rides the same trunk interface into the router’s bridge. From the live RouterOS config:
vlan-id 8 lab-int (management uplink to the switch)
vlan-id 9 bmc (server BMC/IPMI network)
vlan-id 11 mgm (OSA management network, br-mgmt)
vlan-id 12 vxlan (OSA tunnel network, VXLAN overlay)
vlan-id 13 k8s-a (k8s cluster A VLAN)
vlan-id 14 project-a (OpenStack tenant network A)
vlan-id 15 project-b (OpenStack tenant network B)
vlan-id 16 k8s-b (k8s cluster B VLAN)
vlan-id 18 os-net-bgp (OpenStack provider network used for BGP peering)
vlan-id 19 lbaas (reserved for Octavia LBaaS, not yet enabled)
On the OpenStack-Ansible side, the provider VLAN range for tenant networks is configured as a contiguous block rather than the whole 4094-VLAN space — deliberately narrow, matching only what’s actually wired to the switch:
# openstack_user_config.yml
- network:
container_bridge: "br-vlan"
type: "vlan"
range: "14:18" # tenant VLANs; 19 is reserved for lbaas, 20 storage, 21 osd
net_name: "provider"3. BGP: how the router talks to each cluster’s CNI
Every Kubernetes cluster and the OpenStack tenant network peers with the router over eBGP, not static routes. The AS numbering follows one rule consistently: router-side ASN = node-side ASN + 2:
| Peering | Router AS | Node AS |
|---|---|---|
| OpenStack tenant (compute hosts) | 65014 | 65015 |
| k8s project a CNI | 65023 | 65021 |
| k8s project b CNI | 65027 | 65025 |
Each BGP instance gets its own router-id and its own accept-only-this-prefix filter chain — nothing is accepted by default, each peering can only advertise the one CIDR block it’s actually supposed to own:
/routing bgp instance add as=65023 name="k8s project a cni" router-id=10.14.0.1 routing-table=main
/routing bgp instance add as=65014 name="OS tenant" router-id=10.12.0.1 routing-table=main
/routing bgp instance add as=65027 name="k8s project b cni" router-id=10.15.0.1 routing-table=main
# one connection per node — example, k8s-a masters/nodes:
/routing bgp connection add as=65023 instance="k8s project a cni" local.address=10.14.0.1 \
remote.address=10.14.0.21 .as=65021 templates=k8s-a-cni-template name=k8s-a-master-1
# ... repeated per node (10.14.0.21-23 masters, .31-.36 workers)
# the filter is the real security boundary — only this cluster's own service/pod CIDRs are accepted:
/routing filter rule add chain=k8s-a-in rule="if (dst in 10.13.0.0/18 || dst in 10.13.64.0/18 || dst in 10.13.128.0/18) { accept }"
/routing filter rule add chain=k8s-a-out rule=reject # never advertise anything backThe OpenStack tenant peering additionally uses tcp-md5-key on the BGP session (the compute hosts sit across VLANs from the router, multihop=yes) — the k8s CNI peerings don’t need it since Calico’s own BGP password support via a Kubernetes Secret covers that layer instead.
4. Installing OpenStack (OpenStack-Ansible)
Standard 3-node OSA, deployed from a bootstrap/deploy host with SSH access to all 3 physical servers.
4.1 Base OS prep
Before OSA touches anything, every target host needs to match OpenStack-Ansible’s own “prepare the target hosts” prerequisites: short (non-FQDN) hostnames — long names blow past LXC/SSH naming limits — the en_US.UTF-8 locale, SSH key-based auth with passwordless sudo, and the base package set OSA’s docs call for (bridge-utils, lvm2, openssh-server, tcpdump, python3). The docs also call for four network bridges: br-mgmt on every host (OSA’s own management/API traffic), br-vxlan for the tunnel network, br-vlan for VLAN/flat provider networks, and br-storage for Cinder/Ceph traffic.
On top of that baseline, this lab’s own host-prep adds: openvswitch-switch (the bridges above are OVS bridges here, not Linux bridges — see the lxc_net_bridge_type override in §5), ceph-volume (these hosts double as Ceph OSD nodes), KVM kernel modules auto-detected from CPU vendor, and persistent iptables rules.
4.2 Bootstrap Ansible itself, then generate secrets
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
scripts/bootstrap-ansible.sh
# auto-generate every OpenStack service password/token — never hand-write these
cd /opt/openstack-ansible/scripts/
python pw-token-gen.py --file /etc/openstack_deploy/user_secrets.yml4.3 The 3 real install playbooks
cd /opt/openstack-ansible/playbooks
openstack-ansible setup-hosts.yml # 1. prepare hosts + LXC containers
openstack-ansible setup-infrastructure.yml # 2. galera, rabbitmq, memcached, internal haproxy
openstack-ansible setup-openstack.yml # 3. keystone, nova, neutron, cinder, glance, horizon...A real gotcha hit during this install: instances had no internet connectivity until this was set:
sed -i 's|is_metal:.*|is_metal: false|g' /opt/openstack-ansible/inventory/env.d/neutron.yml5. Non-default configuration (openstack_user_config.yml / user_variables.yml)
What actually deviates from OSA’s defaults in this deployment:
- Network CIDRs — management
10.11.0.0/24, tunnel10.12.0.0/24, storage10.80.0.0/24(LBaaS reserved at10.19.0.0/24but not enabled yet) - HAProxy VIPs — internal
10.11.0.10, external10.11.0.9— these must resolve to an address insidehaproxy_keepalived_external_vip_cidr - Combined host roles — all 3 physical hosts (
infra1/2/3) serve as infrastructure, compute, identity, storage-infra, load-balancer and repo hosts simultaneously — there’s no dedicated controller tier, deliberate for a 3-node cloud - Ceph as the Cinder/Glance backend, not LVM — the LVM backend block exists in the config but is commented out
- VXLAN tunnel range
1:1000, tenant VLAN range14:18(see §2) - Tuning:
galera_innodb_buffer_pool_size: 4G,galera_max_connections: 1000,haproxy_maxconn: 4096, LXC bridge on Open vSwitch (lxc_net_bridge_type: openvswitch) rather than the Linux-bridge default
6. HAProxy — fronting the OpenStack APIs
This is OSA’s own internal HAProxy (deployed by setup-infrastructure.yml‘s haproxy-install.yml, running on the same 3 load_balancer_hosts). It fronts Keystone, Nova, Neutron, Cinder, Glance, Horizon etc. behind the two VIPs from §5, load-balancing across the 3 infra containers running each service.
Summary of the full chain
- Physical trunk + MikroTik VLANs (§2)
- BGP peering per cluster, filtered to exactly its own CIDR (§3)
- OpenStack-Ansible: bootstrap → setup-hosts → setup-infrastructure → setup-openstack (§4)
- OSA’s internal HAProxy fronts the OpenStack API VIPs, and the cloud is ready for workloads (§6)
