Big Data, Lakehouse & Analytics Cluster Services
Spark, Flink, Trino and an open table format on storage you control — so analytics stops being a separate, expensive, fragile universe.
What this usually looks like
You have data in four places, three of them moving, and a warehouse bill that grows faster than the data does.
- The Hadoop cluster still runs, still nobody upgrades it, and the vendor wants a renewal.
- Analytics jobs starve the production cluster when they run.
- Nobody agrees which table is authoritative.
- Storage and compute are welded together, so scaling one forces scaling the other.
Three ways to buy this
Audit if you want to know how bad it is. Build if you have the team and want the capability. Managed if you don't want to own the pager. Most engagements start with the audit and decide afterwards.
Audit
A fixed-price health check — the cheapest way in
- Current-state inventory: HDFS, warehouses, ingestion paths
- Cost-per-query and cost-per-terabyte review
- Metadata and governance gap analysis
- Migration feasibility for legacy HDFS/Hive estates
Build
One-off delivery: design, code, hardening, handover
- Storage/compute separation: object storage tier + elastic compute on Kubernetes
- Spark on Kubernetes with dynamic allocation, or Spark on YARN where the estate requires it
- Flink session and application clusters for streaming
- Trino/Presto as the federated query layer across sources
- Open table formats: Iceberg, Hudi or Delta on top of MinIO/Ceph/S3
- Orchestration and governance: Airflow/Argo Workflows, DataHub/OpenMetadata
Managed
Build plus 7×24 on-call, SLA and quarterly drills
- Job failure triage and automatic retry policy
- Cost and slot-utilisation reporting per team
- Platform upgrades across Spark/Flink/Trino versions
- Metadata catalogue maintenance and lineage health
What we actually deploy
Specific tooling, not categories. If a version matters to your decision, ask — we will tell you which one we would pick and why, including when the answer is the older one.
Batch compute
- Spark 3.5 on Kubernetes
- Spark on YARN
- MapReduce (legacy)
- Ray Data
Stream processing
- Flink (session + application mode)
- Spark Structured Streaming
- Kafka Streams
- ksqlDB
Query engine
- Trino
- Presto
- Dremio (evaluation)
- DuckDB for local analytics
Table formats & lakehouse
- Apache Iceberg
- Hudi
- Delta Lake
- Project Nessie
- Hive Metastore HA
- AWS Glue-compatible catalog
Legacy maintenance
- HDFS HA
- YARN Federation
- Hive Metastore
- HBase
- Oozie → Airflow migration
Orchestration & governance
- Airflow
- Argo Workflows
- Dagster
- DataHub
- OpenMetadata
- Great Expectations
- dbt (integration)
What we will be measured on
These are the numbers we put in the delivery report. If we cannot measure it, we do not promise it.
- Storage and compute scale independently — proven by resizing one without touching the other.
- Job failures retry per policy and alert with a diagnosis, not just a stack trace.
- Cost per terabyte and cost per query reported per team, per month.
- Migrated legacy tables verified row-count and checksum-equivalent at cutover.
- A single authoritative catalogue with lineage your analysts can actually follow.
What we would build, and where the weak point is
The metastore is the quiet single point of failure in almost every lakehouse. Lose it and every table becomes an unreadable pile of Parquet with no schema history.
marks a single point of failure we would raise with you rather than hide. Every architecture has one; the question is whether you know where it is.
What we don't do
Every vendor says yes to everything. Here is where our answer is no, or where someone else needs to be in the room. If this list looks wrong for your situation, tell us — we would rather argue about scope before the contract than after.
- Out of scope
Data modelling, BI dashboard development, KPI definition
We deliver the platform, catalogue and lineage. What the numbers should mean is your business question.
- Out of scope
Data science and model development
We provide the compute. See our HPC & AI line for GPU-based training and serving.
- Requires partner
Cloud warehouse licences and their billing disputes
We can operate Snowflake/BigQuery/Databricks alongside your own stack; we do not resell or negotiate their contracts.
- Out of scope
Guaranteeing "single source of truth" across organisations
That is a governance decision, not an engineering deliverable. We build the tooling that makes the decision enforceable.
What it looked like last time
A media company moved a Spark/Hive estate onto Iceberg on Ceph with Trino in front: repeated nightly jobs dropped from 6.5 hours to 48 minutes, and the storage bill fell because compute no longer scaled with retention.
Anonymised under NDA. No client names, no logos, no invented metrics. Full case library on the work page.
Talk to the data platform engineer about your big data & lakehouse
Send us the architecture, or just describe what broke most recently. You'll get an engineer's answer, not a discovery questionnaire.
Talk to a data platform engineer: botonwa@gmail.com · +86 130 5181 3099