The 80km Layer 2 Problem That Forced Our Hand
We needed to extend a tenant’s Layer 2 segment across two data centers 80km apart, and traditional VLAN extension over the WAN was not viable at the bandwidth cost. The manufacturing application behind it was ugly in the way old industrial systems often are: hard-coded gateways, license binding to subnets, and a maintenance window measured in minutes because the plant floor did not care about our network architecture preferences.
My first thought was to stretch the VLAN with the tools we already had, then police it hard at the edge. That was the wrong first assumption. Once we modeled ARP, unknown unicast, and multicast behavior across the inter-DC link, the design looked cheap only on the whiteboard.
We were running FortiOS 7.4.3 at the security edge, Ubuntu 22.04 jump hosts for validation, and Python 3.11 scripts for packet sampling and flow rollups. The switching fabric supported VXLAN termination on the leaf layer, so we moved the tenant into a VXLAN overlay and mapped the old VLAN into a VNI rather than dragging the VLAN itself across the WAN.
Layer 2 stretch is never free.
The part I like about VXLAN is that it admits the real problem: VLAN IDs are too small for modern segmentation. The 12-bit VLAN field gives us 4,094 usable VLANs, while the 24-bit VXLAN Network Identifier gives us roughly 16 million possible segments. In my environment, that meant tenant isolation could grow without turning the VLAN database into a shared spreadsheet full of exceptions, and I prefer designs that reduce human bookkeeping.
How I Mapped VLANs Into VNI Architecture
VXLAN wraps the original Ethernet frame in UDP, normally using destination port 4789, and carries it over the routed underlay. The endpoint doing that encapsulation is the VTEP, and that detail matters because VTEPs become the boundary between the tenant’s Layer 2 world and the routed fabric we can actually operate at distance.
For the tenant extension, we mapped local VLAN 312 to VNI 10312 in both data centers. I kept the numbering obvious because incident response at 2:00 a.m. is not the time to decode clever abstractions. The underlay used routed links, equal-cost paths, and explicit MTU validation because VXLAN adds overhead, and fragmentation across an inter-DC path is a self-inflicted outage.
- VLAN 312 stayed local to each data center leaf pair.
- VNI 10312 became the tenant segment identifier across sites.
- VTEP loopbacks were advertised in the routed underlay.
- MTU was set to 9216 on fabric links and validated end to end.
- Tenant firewalls enforced north-south and inter-segment policy.
Simple identifiers beat clever ones.
The 24-bit VNI space did not make segmentation secure by itself. It gave us scale and clean separation, but the enforcement still lived in routing policy, firewall rules, endpoint controls, and continuous verification. In manufacturing networks, I do not trust labels unless I can test what they block.
Why Flood-and-Learn Looked Fine Until It Wasn’t
Our initial VXLAN deployment used flood-and-learn mode. At low VM counts, it behaved well enough that nobody complained. VTEPs learned remote MAC addresses from data-plane traffic, and unknown unicast, broadcast, and multicast traffic crossed the overlay when the fabric had no better answer.
Then the tenant grew past 500 VMs per VNI. BUM traffic started consuming 18% of inter-DC bandwidth unnecessarily, and that was before a scheduled application restart caused a wave of ARP and service discovery traffic. The WAN graph made the mistake obvious. I had treated flood-and-learn as a temporary shortcut, then allowed it to survive long enough to become production architecture.
What I didn’t expect was how quiet the failure looked from the application side. Nothing went fully down. Latency got inconsistent, replication jobs ran longer, and the security tools saw noisy east-west chatter that was technically allowed but operationally wasteful.
Waste hides inside “working.”
Moving to BGP EVPN changed the behavior because MAC and IP reachability moved into the control plane. Instead of flooding to discover where endpoints lived, VTEPs exchanged EVPN routes and made forwarding decisions with intent. After migrating to BGP EVPN control plane, BUM traffic reduced from 18% to 2% of inter-DC bandwidth, which was the exact metric that convinced our operations manager the added control-plane complexity was justified.
You may also find this useful: Check out our guide on Python Network Config Backup: Automating Multi-Vendor Device Snapshots for more practical tips.
What My VTEP Build Looked Like
The exact syntax changes by vendor, but the pattern stayed consistent across our build: stable loopbacks, routed underlay reachability, VXLAN interface creation, VLAN-to-VNI mapping, and EVPN peering. I tested from an Ubuntu 22.04 jump host using Python 3.11 flow parsers, packet captures, and CLI checks before I let the tenant ride the overlay during a maintenance window.
# Validate underlay reachability and VXLAN MTU from a jump host
ping -M do -s 8972 10.40.0.12
tracepath 10.40.0.12
# Check EVPN route exchange on a network OS with vtysh-style CLI
vtysh -c "show bgp l2vpn evpn summary"
vtysh -c "show bgp l2vpn evpn route vni 10312"
vtysh -c "show evpn mac vni 10312"
# Capture VXLAN traffic for the tenant VNI
sudo tcpdump -ni ens192 udp port 4789 -vv
I also forced endpoint movement between data centers and watched MAC mobility updates. That test mattered more than a clean ping because the real operational risk was not basic reachability; it was stale forwarding after a migration event. If the control plane could not converge cleanly, the application team would see it before my dashboard did.
Trust the failure test.
On the firewall side, FortiOS 7.4.3 gave us the policy enforcement point we needed after decapsulation and routing. I did not want the overlay to become a bypass around segmentation just because two workloads shared a VNI. In my opinion, VXLAN should extend a segment only when the business case is painful enough to justify the operational burden.
Verify Tenant Isolation Before You Trust the Overlay
I treated tenant isolation as something to prove, not something to infer from configuration. We checked that VNI 10312 could not leak into adjacent VNIs, that default gateways stayed tenant-scoped, and that firewall logs showed denied attempts where we expected denial. Packet captures confirmed VXLAN headers, but policy tests confirmed security.
My team built a small verification routine that compared intended segmentation against observed flows. It pulled firewall logs, sampled NetFlow, and flagged any source-destination pair outside the allowed matrix. The first run caught a legacy monitoring exception that crossed tenant boundaries because someone had copied an old rule group during migration. That was uncomfortable, and it was useful.
Configuration is not evidence.
The operational lesson for me was blunt: VXLAN gives us a larger segmentation namespace and a practical way to carry Layer 2 across routed infrastructure, but BGP EVPN is what makes that design sane at scale. Flood-and-learn may be acceptable in a lab or a tiny segment, yet I would not run it for a production tenant with hundreds of workloads across data centers again.
I now require three checks before approving this pattern in our environment: EVPN control plane enabled, BUM traffic measured before and after migration, and tenant isolation verified with live traffic tests. That standard is stricter than some vendor reference designs, and I am fine with that.