{"id":156,"date":"2026-08-17T00:49:56","date_gmt":"2026-08-17T00:49:56","guid":{"rendered":"http:\/\/127.0.0.1\/en\/docs\/kubernetes-on-openstack-terraform-kubespray-cilium-bgp\/"},"modified":"2026-08-17T00:56:54","modified_gmt":"2026-08-17T00:56:54","slug":"kubernetes-on-openstack-terraform-kubespray-cilium-bgp","status":"publish","type":"post","link":"https:\/\/wp-nv.radut.info\/en\/2026\/08\/17\/kubernetes-on-openstack-terraform-kubespray-cilium-bgp\/","title":{"rendered":"Kubernetes on OpenStack: Terraform, Kubespray &#038; Cilium BGP"},"content":{"rendered":"<p><strong>Scope:<\/strong> this is the layer that sits on top of the OpenStack cloud described in <em>OpenStack Lab Deployment<\/em> \u2014 how Kubernetes clusters get provisioned as OpenStack VMs with Terraform, installed with Kubespray, and wired into the same MikroTik router via Cilium&#8217;s BGP control plane (plus L2 announcements) instead of NodePort\/NAT tricks. Real values from this deployment (ASNs, CIDRs, CRDs) are included throughout; credentials and private keys are intentionally omitted.<\/p>\n<h2>1. Architecture at a glance<\/h2>\n<p>Two Kubernetes clusters live on top of the same OpenStack cloud, each on its own provider VLAN (see the sibling article&#8217;s VLAN table): <strong>k8s project a<\/strong> and <strong>k8s project b<\/strong>. Both are provisioned the same way: Terraform creates the OpenStack VMs (Nova instances, ports on the cluster&#8217;s provider network, Cinder volumes), then Kubespray installs Kubernetes on top over SSH. Cilium is the CNI on both, in <code>kube-proxy<\/code>-replacement mode, with its BGP control plane peering directly with the MikroTik router \u2014 no external LoadBalancer, no NodePort\/NAT \u2014 pod and Service IPs get real routes.<\/p>\n<h2>2. Provisioning the VMs with Terraform, then Kubespray<\/h2>\n<p>Each cluster has its own Terraform workspace (Nova instances sized from the flavor list, ports on that cluster&#8217;s provider VLAN, a Cinder boot volume per node) feeding a generated Ansible inventory that Kubespray installs against. The real post-install sequence, run once the nodes are up and Kubespray has finished:<\/p>\n<pre><code>bash config-kubectl.sh\nbash script-cni.sh              # Cilium: helm install + BGP\/LB-IP-pool CRDs (see \u00a73-5)\nbash script-ingress-haproxy.sh  # ingress-nginx\/HAProxy ingress controller\nbash script-cert-manager.sh     # see \u00a77\nbash script-dns-cm-autoscaler.sh\nbash script-registry-secret.sh\nbash script-csi-setup.sh        # Ceph RBD\/CephFS CSI, backed by the same OpenStack-hosted Ceph<\/code><\/pre>\n<p>The ingress HAProxy here is a completely different deployment from the OSA-internal HAProxy in the sibling article \u2014 that one fronts the OpenStack control-plane APIs (Keystone, Nova, etc.); this one is a Kubernetes <code>IngressClass<\/code> controller fronting workloads running inside the cluster.<\/p>\n<h2>3. Cilium install<\/h2>\n<p>Both clusters run the same chart, same major settings, different only in their per-cluster ASN\/CIDR:<\/p>\n<pre><code>helm upgrade --install cilium oci:\/\/quay.io\/cilium\/charts\/cilium --version 1.19.1 \\\n  --namespace kube-system -f cni\/cilium\/cilium-values.yaml --wait --timeout=5m\n\nkubectl apply -f cni\/cilium\/crds\/bgp-cluster-config.yaml\nkubectl apply -f cni\/cilium\/crds\/bgp-peer-config.yaml\nkubectl apply -f cni\/cilium\/crds\/bgp-advertisement.yaml\nkubectl apply -f cni\/cilium\/crds\/loadbalancer-ip-pool.yaml<\/code><\/pre>\n<p>Non-default values on top of the chart&#8217;s defaults:<\/p>\n<ul>\n<li><code>kubeProxyReplacement: true<\/code>, <code>kube_proxy_remove: true<\/code> \u2014 Cilium fully replaces kube-proxy<\/li>\n<li><code>autoDirectNodeRoutes: true<\/code>, tunnel mode disabled \u2014 routing is direct, not VXLAN-encapsulated, since the underlying OpenStack provider VLAN already gives L2 reachability between nodes<\/li>\n<li><code>bgpControlPlane.enabled: true<\/code> \u2014 the CRD-driven BGP control plane (\u00a74-5), the actual mechanism used to reach the MikroTik router<\/li>\n<li><code>l2announcements.enabled: true<\/code> on k8s project a&#8217;s Helm values (\u00a76) \u2014 see that section for the honest caveat<\/li>\n<li><code>ipam.mode: kubernetes<\/code>, <code>enable-ipv4-masquerade: true<\/code> with an explicit non-masquerade CIDR list (<code>10.0.0.0\/8<\/code>, <code>172.16.0.0\/12<\/code>, <code>192.168.0.0\/16<\/code>) so traffic to any RFC1918 range \u2014 including the other cluster and the OpenStack tenant networks \u2014 isn&#8217;t SNAT&#8217;d<\/li>\n<\/ul>\n<h2>4. BGP: Cilium \u2194 MikroTik peering<\/h2>\n<p>Same AS-numbering convention as the OpenStack article: <strong>router-side ASN = node-side ASN + 2<\/strong>. Cilium peers directly with the MikroTik&#8217;s per-cluster BGP instance from the sibling article&#8217;s \u00a73 \u2014 no OpenStack-side BGP dragent\/Neutron BGP speaker involved, this is a straight eBGP session from each node to the router IP on that cluster&#8217;s provider VLAN:<\/p>\n<table>\n<tr>\n<th>Cluster<\/th>\n<th>Cilium instance<\/th>\n<th>Local ASN (k8s)<\/th>\n<th>Peer (router)<\/th>\n<th>Peer ASN<\/th>\n<\/tr>\n<tr>\n<td>k8s project a<\/td>\n<td><code>instance-65021<\/code><\/td>\n<td>65021<\/td>\n<td>10.14.0.1<\/td>\n<td>65023<\/td>\n<\/tr>\n<tr>\n<td>k8s project b<\/td>\n<td><code>instance-65025<\/code><\/td>\n<td>65025<\/td>\n<td>10.15.0.1<\/td>\n<td>65027<\/td>\n<\/tr>\n<\/table>\n<p><code>CiliumBGPClusterConfig<\/code> (k8s project b shown; project a is identical with the ASNs\/IP above swapped in):<\/p>\n<pre><code>apiVersion: cilium.io\/v2\nkind: CiliumBGPClusterConfig\nmetadata:\n  name: cilium-bgp\nspec:\n  bgpInstances:\n  - name: instance-65025\n    localASN: 65025\n    localPort: 179\n    peers:\n    - name: 10.15.0.1\n      peerASN: 65027       # the MikroTik\n      peerAddress: 10.15.0.1\n      peerConfigRef:\n        name: cilium-peer<\/code><\/pre>\n<p><code>CiliumBGPPeerConfig<\/code> \u2014 the timers and multihop setting are the same on both clusters:<\/p>\n<pre><code>apiVersion: cilium.io\/v2\nkind: CiliumBGPPeerConfig\nmetadata:\n  name: cilium-peer\nspec:\n  timers:\n    connectRetryTimeSeconds: 5\n    holdTimeSeconds: 15\n    keepAliveTimeSeconds: 5\n  gracefulRestart:\n    enabled: true\n    restartTimeSeconds: 20\n  ebgpMultihop: 5\n  transport:\n    peerPort: 179\n  families:\n  - afi: ipv4\n    safi: unicast\n    advertisements:\n      matchLabels:\n        advertise: bgp<\/code><\/pre>\n<p><code>CiliumBGPAdvertisement<\/code> \u2014 pod CIDR and every LoadBalancer\/ExternalIP Service address get advertised, tagged with a BGP community per ASN so the router-side filter chain (sibling article \u00a73) can still enforce &#8220;only this cluster&#8217;s own prefixes&#8221;:<\/p>\n<pre><code>apiVersion: cilium.io\/v2\nkind: CiliumBGPAdvertisement\nmetadata:\n  name: bgp-advertisements\n  labels:\n    advertise: bgp\nspec:\n  advertisements:\n  - advertisementType: PodCIDR\n    attributes:\n      communities:\n        standard: [\"65026:100\"]\n  - advertisementType: Service\n    attributes:\n      communities:\n        standard: [\"65025:100\"]\n    service:\n      addresses:\n      - LoadBalancerIP\n      - ExternalIP        # only if you need manual static IP control\n    selector:\n      matchExpressions:\n      - {key: somekey, operator: NotIn, values: [\"never-used-value\"]}   # empty selector = advertise all LB Services<\/code><\/pre>\n<p>ClusterIP Services are deliberately never advertised over BGP \u2014 pod-to-pod and pod-to-ClusterIP traffic stays on Cilium&#8217;s own eBPF datapath; advertising ClusterIPs externally would defeat the point of them being cluster-internal.<\/p>\n<h2>5. LoadBalancer IP pools<\/h2>\n<p>Each cluster gets its own <code>CiliumLoadBalancerIPPool<\/code>, carved out of that cluster&#8217;s own pod\/service supernet (also the block accepted by the router-side filter in the sibling article):<\/p>\n<table>\n<tr>\n<th>Cluster<\/th>\n<th>Native routing CIDR<\/th>\n<th>LoadBalancer IP pool<\/th>\n<\/tr>\n<tr>\n<td>k8s project a<\/td>\n<td>10.13.0.0\/16<\/td>\n<td>10.13.128.0\/18<\/td>\n<\/tr>\n<tr>\n<td>k8s project b<\/td>\n<td>10.16.0.0\/16<\/td>\n<td>10.16.128.0\/18<\/td>\n<\/tr>\n<\/table>\n<pre><code>apiVersion: cilium.io\/v2\nkind: CiliumLoadBalancerIPPool\nmetadata:\n  name: default-pool\nspec:\n  allowFirstLastIPs: \"No\"\n  blocks:\n  - cidr: 10.16.128.0\/18   # k8s project b; 10.13.128.0\/18 on k8s project a<\/code><\/pre>\n<h2>6. L2 announcements<\/h2>\n<p>L2 announcements and BGP solve different problems and are normally either\/or per environment: L2 announcements make a Service&#8217;s external IP answer ARP\/NDP on the local segment (useful when there&#8217;s no upstream router able to do BGP), while BGP gets the IP a real route into the network. Since the MikroTik here does speak BGP, BGP (\u00a74) is the actual mechanism carrying traffic on both clusters.<\/p>\n<p><strong>Honest caveat:<\/strong> k8s project a&#8217;s Helm values do set <code>l2announcements.enabled: true<\/code> at the chart level (<code>cilium_l2announcements: true<\/code> in that cluster&#8217;s Kubespray inventory), but no <code>CiliumL2AnnouncementPolicy<\/code> custom resource exists anywhere in this repo for either cluster. Enabling the feature flag alone doesn&#8217;t announce anything \u2014 without a policy CR scoping which Services\/interfaces to announce, this is a no-op today, effectively dormant. It&#8217;s flagged here as a real gap rather than glossed over: turning it into an active fallback (for example, restricted to a specific interface on the provider VLAN, only for Services carrying a particular label) is a follow-up, not something already running.<\/p>\n<h2>7. cert-manager + Route53 DNS-01<\/h2>\n<pre><code>helm install cert-manager oci:\/\/quay.io\/jetstack\/charts\/cert-manager \\\n  --version v1.19.4 --namespace cert-manager --create-namespace \\\n  --set crds.enabled=true --wait --timeout=5m<\/code><\/pre>\n<p>DNS-01 via Route53 rather than HTTP-01 \u2014 one wildcard certificate covers every vhost in a single order, and it works for hosts that only resolve on the internal LAN:<\/p>\n<pre><code># 1. Secret holding the AWS secret access key (never commit this value)\nkubectl -n cert-manager create secret generic route53-credentials \\\n  --from-literal=secret-access-key='&lt;AWS_SECRET_ACCESS_KEY&gt;'\n\n# 2. ClusterIssuer referencing it \u2014 access key ID is not secret, secret key is\napiVersion: cert-manager.io\/v1\nkind: ClusterIssuer\nmetadata:\n  name: letsencrypt-dns\nspec:\n  acme:\n    server: https:\/\/acme-v02.api.letsencrypt.org\/directory\n    email: you@example.com\n    privateKeySecretRef: { name: letsencrypt-dns-account-key }\n    solvers:\n      - dns01:\n          route53:\n            region: us-east-1\n            accessKeyID: &lt;AWS_ACCESS_KEY_ID&gt;\n            secretAccessKeySecretRef:\n              name: route53-credentials\n              key: secret-access-key\n---\n# 3. One wildcard Certificate covers every host under the domain\napiVersion: cert-manager.io\/v1\nkind: Certificate\nmetadata:\n  name: example-wildcard\n  namespace: haproxy-controller\nspec:\n  secretName: example-wildcard-tls\n  issuerRef: { name: letsencrypt-dns, kind: ClusterIssuer }\n  commonName: \"*.example.com\"\n  dnsNames: [\"example.com\", \"*.example.com\"]<\/code><\/pre>\n<p>The wildcard secret is set as the ingress controller&#8217;s <code>defaultTLSSecret<\/code>, so every <code>Ingress<\/code> resource gets TLS with no per-host <code>tls:<\/code> block. The IAM user backing this only needs <code>route53:ChangeResourceRecordSets<\/code>\/<code>GetChange<\/code>\/<code>ListResourceRecordSets<\/code> scoped to the one hosted zone.<\/p>\n<h2>Summary of the full chain<\/h2>\n<ol>\n<li>OpenStack up and its provider VLANs routed via BGP (sibling article)<\/li>\n<li>Terraform provisions the cluster&#8217;s VMs on that cluster&#8217;s own provider VLAN<\/li>\n<li>Kubespray installs Kubernetes over SSH against the generated inventory<\/li>\n<li>Cilium: kube-proxy replacement, BGP control plane peering to the MikroTik with the same router-AS-plus-2 convention (\u00a73-4)<\/li>\n<li>Per-cluster LoadBalancer IP pool, carved from that cluster&#8217;s own routed CIDR (\u00a75)<\/li>\n<li>L2 announcements flagged on but not yet backed by a policy \u2014 dormant, noted as a gap (\u00a76)<\/li>\n<li>Ingress HAProxy + cert-manager + Route53 DNS-01 (\u00a77), Ceph CSI, and the cluster is ready for workloads<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Scope: this is the layer that sits on top of the OpenStack cloud described in OpenStack Lab Deployment \u2014 how Kubernetes clusters get provisioned as&#8230;<\/p>\n<div class=\"more-link-wrapper\"><a class=\"more-link\" href=\"https:\/\/wp-nv.radut.info\/en\/2026\/08\/17\/kubernetes-on-openstack-terraform-kubespray-cilium-bgp\/\">Continue reading<span class=\"screen-reader-text\">Kubernetes on OpenStack: Terraform, Kubespray &#038; Cilium BGP<\/span><\/a><\/div>\n","protected":false},"author":0,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"qubely_global_settings":"","qubely_interactions":"","_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","_uag_custom_page_level_css":"","footnotes":""},"categories":[21],"tags":[],"class_list":["post-156","post","type-post","status-publish","format-standard","hentry","category-openstack","entry"],"qubely_featured_image_url":null,"qubely_author":{"display_name":"","author_link":"https:\/\/wp-nv.radut.info\/en\/author\/"},"qubely_comment":0,"qubely_category":"<a href=\"https:\/\/wp-nv.radut.info\/en\/category\/openstack\/\" rel=\"category tag\">OpenStack<\/a>","qubely_excerpt":"Scope: this is the layer that sits on top of the OpenStack cloud described in OpenStack Lab Deployment \u2014 how Kubernetes clusters get provisioned as&#8230;Continue readingKubernetes on OpenStack: Terraform, Kubespray &#038; Cilium BGP","uagb_featured_image_src":{"full":false,"thumbnail":false,"medium":false,"medium_large":false,"large":false,"1536x1536":false,"2048x2048":false,"trp-custom-language-flag":false,"qubely_landscape":false,"qubely_portrait":false,"qubely_thumbnail":false},"uagb_author_info":{"display_name":"","author_link":"https:\/\/wp-nv.radut.info\/en\/author\/"},"uagb_comment_info":0,"uagb_excerpt":"Scope: this is the layer that sits on top of the OpenStack cloud described in OpenStack Lab Deployment \u2014 how Kubernetes clusters get provisioned as&#8230;Continue readingKubernetes on OpenStack: Terraform, Kubespray &#038; Cilium BGP","_links":{"self":[{"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/posts\/156","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/comments?post=156"}],"version-history":[{"count":1,"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/posts\/156\/revisions"}],"predecessor-version":[{"id":169,"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/posts\/156\/revisions\/169"}],"wp:attachment":[{"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/media?parent=156"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/categories?post=156"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wp-nv.radut.info\/en\/wp-json\/wp\/v2\/tags?post=156"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}