Retiring OPNsense for a UniFi Cloud Gateway

Problem

For years the edge of my lab was OPNsense, and OPNsense was a virtual machine on the hypervisor it protected. It worked, and it taught me more about firewalls, routing and DNS than any course would have. It also had a dependency loop hiding in it: the hypervisor reached the network through a switch, and the switch reached the internet through a virtual machine living on that hypervisor. Nothing about that is wrong until the day the host needs to come up alone, and then the order in which things boot stops being an academic question.

There was a second problem underneath the first, and it was older. The services network rode the switch’s native VLAN, VLAN 1, because that is what it defaults to and nobody had ever gone back and fixed it. Every host, every virtual machine and every camera on the most important segment of the lab was sitting on the one VLAN you are told never to use.

OPNSENSE, THE VM THAT WAS THE ROUTER

So the job was two migrations that had to stay separate: move the edge onto a dedicated UniFi Cloud Gateway Fiber, then move the services network onto a real tagged VLAN. Doing both in one window would have meant a failure could come from either one, with vCenter and the entire virtual estate on the other side of the guess.

Approach

I did recon first and changed nothing, which produced a blueprint of the running network rather than a memory of it: every DHCP reservation, every local DNS record, every firewall rule, the port map, and which hosts were static versus leased. That last question mattered more than any other. vCenter and ESXi turned out to be statically addressed, proven by the fact that neither had ever produced a DHCP acknowledgement, which collapsed the risk to the virtual estate down to two DNS records.

Two invariants then protected everything else. The gateway keeps its address and no subnet changes. Every host on the lab had that address baked in as its default route and its resolver, so holding it meant nothing behind the firewall had to be reconfigured, or even know a migration had happened. The old WAN interface’s MAC address was cloned onto the new one so the provider handed back the same public address rather than treating it as a new subscriber.

The new gateway was built on an isolated bench with its WAN deliberately unplugged, and all 104 records, 53 address reservations and 51 local DNS entries, were loaded by a script driving the console’s own session, dry run by default. Recursive DNS carried a subtlety with it: the old resolver had been made authoritative for the lab’s internal subdomains so that search-suffixed lookups returned nothing instead of falling through to a public wildcard and resolving to my own WAN address. The replacement had to be authoritative in exactly the same way, or a bug I had already fixed once would come straight back.

Every segment is a tagged VLAN with its own subnet and its own scope, and the top row is where the second phase ended up. VLAN 1 sits parked on a subnet nothing uses, DHCP off, not a single lease against it. It is still listed because a UniFi gateway will not let you delete its default network, but nothing rides it any more.

Two findings from the switch changed the plan, and both would have taken the lab down at the worst possible moment. The uplink port carried BPDU guard, which was harmless for years because OPNsense never emitted a spanning-tree frame; a UniFi gateway’s integrated switch does, and the port would have error-disabled itself the instant the new edge came up. And the core switch was root bridge only by accident, on default priorities, so the new gateway could have stolen root mid-cutover. I pinned the root priority before touching anything else.

The build sheet was wrong about the most important thing in it, and measuring is what caught it. It called for the services network to be created as a tagged VLAN 10 from day one. The switch was carrying that network untagged on VLAN 1, with zero ports assigned to VLAN 10. Building it as designed would have dropped the entire lab the moment the new gateway took over. It was built untagged, exactly as the network actually was, and the re-tag became its own phase.

Cutover itself was the short part, because everything before it had been about making it short: move the WAN, move the LAN trunk, power the old firewall off without deleting it. Rollback was five minutes and two cables for the whole soak week, which is the only reason a change of that size is a reasonable thing to do on a Saturday.

Result

The lab came back on the other side without noticing. Same public address, same gateway address, same subnets, vCenter green, cameras recording, and remote access still working because the VPN and the tunnel both lived behind the firewall rather than inside it. Every VLAN was carried by the new edge on day one.

10 GBE ON THE PROVIDER PORT · NATIVE VLAN PER PORT

The VLAN re-tag came afterwards and turned out to be the more interesting half. Measurement showed that no endpoint on the network had ever emitted a VLAN 1 tag: VLAN 1 was pure untagged classification that happened at the switch port, not something any host knew about. That meant the whole segment could be moved by re-classifying untagged traffic at each ingress point instead of reconfiguring anything that produced it. Every fixed reservation was re-pointed through the gateway’s API in one pass, every access port was moved, and ESXi, every virtual machine, every camera and every client needed no change at all. Downtime was the seconds each remap took, the management VLAN was never touched, and VLAN 1 now has zero member ports.

What sits at the edge now is a zone-based policy instead of an ordered rule list, default deny between zones, with the VPN segment given a zone of its own so that every future exception is one policy rather than surgery inside the rules that protect everything else. That segment egresses through a commercial VPN provider by policy route, which is a thing I could not have done cleanly on the old edge.

The lesson I actually carry from it has nothing to do with firewalls. A migration plan is a hypothesis, and the running network is the authority. The build sheet had the services VLAN wrong, the uplink port had a guard on it nobody had thought about in years, and the root bridge was correct by luck. None of that was visible in the design; all of it was visible in the device. There is a smaller version of the same lesson in how I nearly reported the finished work as broken: a policy route takes one to three minutes to provision, I tested it at forty seconds, saw it fail, and said so out loud before it had ever had a chance to be true.