A failed node means a site visit
There is no path to recover a site remotely, so a two-hour fix becomes a two-day round trip and a production interruption.
Solutions · Edge fleets
Manufacturing & edge fleets
Dozens of small sites, intermittent connectivity, and nobody on site who can debug a node.
What makes this sector awkward
Sites are numerous and thinly staffed. Connectivity is unreliable by nature, hardware is exposed to environment and to power that is not clean, and the person nearest the equipment is usually a machine operator rather than an engineer.
K3s or KubeVirt at the edge with central fleet management, an out-of-band path that survives the data plane, offline-tolerant updates, and recovery that does not need a truck roll.
Four situations we see constantly
If two or three of these describe your environment, the audit is worth the money. If all four do, it is overdue.
Configuration drift across the fleet is unknown
Sites were installed over years by different people. Nobody can say which of them differ from the reference, or in what way.
Management shares the data path
Remote access depends on the link that also carries production traffic, so when that link has a problem, so does your ability to see it.
Updates are a gamble
An update that assumes a stable link bricks sites when the link drops mid-transfer, and rollback is a truck roll.
Four phases, each useful on its own
Sequenced so that stopping after any phase leaves you better off rather than stranded mid-migration.
- 01
Establish an independent management path
Give every site out-of-band access that does not depend on the production link, so a link failure does not remove your ability to diagnose it.
You end up withA management path that survives a data-plane outage, tested by deliberately breaking the data plane.
- 02
Make drift visible
Bring every site onto a declared configuration and report the ones that deviate, rather than discovering them one outage at a time.
You end up withFleet drift detected and reported centrally, with a remediation path that does not require visiting.
- 03
Make updates interruptible
Stage updates locally, apply them atomically, and keep the previous version so a failed update is an automatic revert rather than a site visit.
You end up withAn update that survives a link drop mid-transfer, demonstrated with the link actually dropped.
- 04
Automate remote recovery
Ensure a site can return itself to a known good state — including after a power event — without anyone on site.
You end up withRemote node recovery without a site visit, tested against a real power interruption.
The capability behind it
Each discipline has its own page with the technology stack, the acceptance criteria and an explicit scope boundary.
Multi-Cluster Fleet Management, Disaster Recovery & Security
The layer that holds every other cluster type together: fleets that deploy the same way everywhere, DR that has been rehearsed, and security that produces evidence.
Karmada · Rancher Fleet · Argo CD ApplicationSet · Cluster APINetwork Cluster Engineering — Firewall, Switch & Router High Availability
Firewall pairs that actually fail over on purpose, switch stacks and MLAG pairs that neither loop nor blackhole when a member dies, and routers whose routing convergence you have measured instead of assumed.
Arista EOS · Cumulus Linux · SONiC · Cisco NX-OS (vPC)Bare Metal Linux Cluster Deployment
Twenty machines arrive. In fifteen minutes each they are identical, provisioned from Git, and ready to join a cluster — with nobody logging in by hand.
PXE / iPXE · Debian Preseed · Kickstart · MAASVirtualization, Private Cloud & VMware Migration Services
Licence costs that do not triple every renewal. Proxmox VE, Ceph, KubeVirt and OpenStack, with a migration plan that includes a tested way back.
Proxmox VE 8.x · XCP-ng · oVirt / OLVM · KVM + libvirt with manual managementHow this looks in practice
Colocation & hosting provider · West Coast
A firewall pair that had never failed over, and 41 devices with no configuration backup
2.8s · measured stateful failover
Read the engagement → Retail chain · North AmericaEighteen sites on one overlay, where a failed link is an event rather than an outage
4 · WAN links per site, all in active use
Read the engagement → Telecommunications software · GlobalTwenty-three clusters, one repository, and drift that gets corrected rather than reported
23 · clusters under one declarative control plane
Read the engagement →Every case in this library is a representative engagement, written from our delivery patterns and marked as such. We do not present unnamed work as a named customer.
What we measure in this sector
These are the numbers we put in the delivery report. If we cannot measure it, we do not promise it.
- Fleet drift detected and remediated centrally
- Update survivable through a link outage, demonstrated
- Remote recovery without a site visit
- Out-of-band path verified independent of the data plane
What we don't do
Every vendor says yes to everything. Here is where our answer is no, or where someone else needs to be in the room. If this list looks wrong for your situation, tell us — we would rather argue about scope before the contract than after.
- Out of scope
OT, PLC and industrial control system work
We build the IT layer beside the plant network. Control systems, safety functions and their certification are a different discipline with different rules.
- Requires partner
Carrier circuits, cellular plans and site connectivity contracts
We design around what the link does. Ordering and holding the circuit is a commercial relationship with a carrier.
- Requires partner
Physical installation, environmental and electrical work
Racking, power and cooling at remote sites needs a local trade. We will specify and verify rather than pretend to install.
- Included
Fleet architecture, out-of-band design, update mechanism and remote recovery
Including the failure tests that prove it works from the head office.
Edge fleets specifics
Our sites have genuinely unreliable power. Can you design for that?
Yes, and it changes the design rather than being an edge case. Read-only boot media, atomic updates, and a recovery path that assumes an unclean shutdown, tested against an actual power cut.
Can you manage sites we cannot reach during business hours?
That is the point of the out-of-band path. We will be explicit about what can be done remotely and what genuinely needs someone on site, so the expectation is set before an incident rather than during one.
How do you handle a site that has been offline for a week?
By design, not by improvisation. The site holds its last known good state, buffers what it can, and reconciles when the link returns. The reconciliation behaviour is tested before deployment, not discovered afterwards.
The engineering transfers, the constraints do not
Financial services & fintech Regulated, audited, and unforgiving of downtime. Every change needs evidence. Healthcare & life sciences Imaging volumes grow faster than budgets, and patient data has strict handling rules. Media & entertainment Render and transcode farms starved by storage, with peaks that double overnight. AI startups & ML teams Expensive accelerators, unstable multi-node training, and no proof of where the time goes. SaaS & internet platforms Multi-tenant isolation, unpredictable growth, and a cloud bill that scales faster than revenue. Research & higher education Slurm on one side, Kubernetes on the other, and a budget committee in the middle.
Want this looked at against your edge fleets environment?
Thirty minutes, no slides. Bring the awkward constraint — that is usually the interesting part.