Distributed storage Representative engagement

An S3 API on your own hardware, with erasure coding and site replication that was actually tested

S3-compatible object storage (MinIO)

Client
Software vendor (on-prem deployment) · US
Region
Two sites, replicated
Scale
8 nodes · 1.6PB usable · 480 million objects
Timeline
9 weeks
Service line
Distributed storage

Where they started

Cloud object storage costs grew with customer retention requirements, and the product needed an on-premises deployment story for customers who could not use a public cloud.

The problem, precisely

The company had to operate object storage they had only ever consumed, including an erasure-coding layout and a replication policy they could defend to their own customers.

What we did

Built for two independent failure domains from the start, tested drive, node and site failure in sequence, and documented the capacity maths including the overhead they were paying.

Delivered

  • Distributed deployment across two sites with an explicit erasure-coding overhead calculation
  • Site replication configured with a measured RPO and a documented failover direction
  • Object Lock and retention policy for regulatory data, tested against deletion attempts
  • Failure tests: one drive, then one node, then one site, each measured
  • Capacity and growth model including EC overhead, so the cost story holds up

Technologies used

  • MinIO (distributed mode)
  • Erasure coding (EC sets)
  • Site replication
  • Bucket policies + Object Lock
  • Prometheus + MinIO metrics
  • Kerberos/LDAP identity integration

Measured result

Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.

  • 480M objects under management
  • 1 site loss survived with no data loss, tested
  • — cost per terabyte per year, modelled against cloud
  • 100% of retention rules enforced by Object Lock, not process
The reusable part

Erasure coding overhead is a first-class number, not a detail. If you cannot state it, you cannot price your own storage.

Something here look familiar?

Most of these started with a cluster audit: two weeks, fixed price, and a risk map that names the single points of failure before you commit to any work. Keep the report either way.