Same engineering, different constraints

You answer to a regulator and an auditor, and both want to see evidence rather than assurances. Availability targets are contractual, recovery objectives are written into agreements, and every production change has to be reconstructable months later by someone who was not there.

Dual-site database topology with a real quorum design, point-in-time recovery that has been restored and timed, and an access model an auditor can follow without a guided tour.

• Measured failover in single-digit seconds, both directions
• Restore-from-backup with a real, quoted RPO
• Access recertification evidence produced on demand
• Zero shared privileged credentials

Imaging and genomics produce data faster than storage budgets grow, the working set is large enough that caching does not help, and the handling rules around patient data constrain where that data can live and who can touch it.

High-throughput storage for imaging, a dual-fabric SAN that has been tested by pulling a path, self-hosted object storage with lifecycle policy, and controls aligned to the framework you are accountable under.

• Storage throughput validated against real imaging workloads, tail included
• Path-failover drill with a measured switchover time
• Retention enforced by the storage layer, reportable as an artifact
• Enforced separation between clinical and research data

Artists and render jobs are only as fast as the storage behind them. Load is spiky and high, files are enormous, and the deadline is usually a release date rather than a quarter. The storage fabric is usually the real ceiling, and it is usually the last thing anybody suspects.

Parallel filesystem for high-bandwidth reads, a storage fabric sized for the real peak rather than the average, a scheduler that packs jobs properly, and burst capacity that does not require a new rack.

• Measured read throughput against the render baseline
• ISL and path latency baselined per fabric
• Peak handled without manual intervention
• Interactive session latency held through batch peaks

The hardware is the most expensive thing in the company and the least measurable. Training runs fail for reasons that look random, inference cost per request is unknown, and the team is under pressure to ship rather than to instrument.

GPU scheduling with gang placement, fabric and storage that keep the accelerators fed, and inference endpoints with a capacity model instead of a guess.

• GPU utilisation measured at the device, before and after
• All-reduce bandwidth baseline at real job sizes
• Requests per second per GPU at the target latency
• Cost per training run, per team

Growth is unpredictable, tenants must be genuinely isolated, and the platform cost is a first-order line in the business model rather than an infrastructure detail. The engineering team is small and every hour spent operating the platform is an hour not spent on the product.

Kubernetes with enforced tenant boundaries, an edge and firewall tier whose failover has been measured, autoscaling that responds to real signals, and cost attribution per team.

• Zero-downtime control-plane and platform upgrades
• Firewall and load-balancer failover tested and timed, not assumed
• Cost per namespace per month, attributable to a team
• Ingress blast radius documented and constrained by admission policy

Two populations with incompatible expectations, funding that arrives in cycles, and equipment that must stay useful for years while the workloads on it change every semester. Fairness is a political property as much as a technical one.

A cluster that serves both scheduler worlds, provisioned from code, with fair-share policy the researchers accept because they were shown the numbers behind it.

• Node bring-up measured in minutes, reproducible from the repository
• Queue time reported per research group
• Fair-share policy documented and defensible with data
• Fleet inside a supported software window

Sites are numerous and thinly staffed. Connectivity is unreliable by nature, hardware is exposed to environment and to power that is not clean, and the person nearest the equipment is usually a machine operator rather than an engineer.

K3s or KubeVirt at the edge with central fleet management, an out-of-band path that survives the data plane, offline-tolerant updates, and recovery that does not need a truck roll.

• Fleet drift detected and remediated centrally
• Update survivable through a link outage, demonstrated
• Remote recovery without a site visit
• Out-of-band path verified independent of the data plane