A machine dies mid-job and the work just sits there.
Every job is held through a renewable lease. When the owner stops renewing, reconciliation detects it and returns the job to a healthy worker, with no one stepping in.
Reliable AI compute, engineered for the day a machine fails.
AetherGrid is an open-source control plane that detects failed workers and recovers their work safely. I help teams building AI infrastructure apply the same discipline.
The orange frame marks the server that owns the job.
The problems your team stops having
AI workloads run for a long time on expensive machines, and machines fail. AetherGrid is built so that the failures your team would otherwise catch by hand are handled by the system.
The same job runs twice, or a late result overwrites a good one.
Each job has one owner at a time. Lease renewal, release and outcome reports are fenced by lease identity, so a stale worker cannot overwrite the current owner.
GPU hours burn on work that nobody needs any more.
When a job finishes, times out, is cancelled or loses its owner, its command is stopped along with anything it left running.
Jobs land on machines that cannot run them well.
Scheduling matches each job to a node on CPU, memory and VRAM together, and a job stays on its own tenant’s nodes. The scheduling policy lives in the domain, where it is easy to read and test.
Several teams or customers share one cluster, and their data must stay apart.
Tenant isolation is enforced in the application and in the database, and the tenant comes from the credential, never from the request.
Upgrades feel risky on a system that has to stay up.
Schema changes ship as versioned migrations. Each release is rehearsed against a database from the release before it, and the upgraded schema is compared with a fresh one.
When something goes wrong, nobody can say what happened.
Every job keeps its event history, requests and background loops write structured logs, and errors are tracked, so an operator can see what happened and why.
A job cannot start, or runs forever, and nobody finds out.
A job whose command cannot start fails with a clear outcome. Each job carries its execution timeout, and a running job can be cancelled.
Why AetherGrid
AI jobs run for a long time on expensive machines, and machines fail. A job that stalls, runs twice or leaves processes behind wastes GPU hours and delays everything waiting on it. AetherGrid makes the system responsible for those failures, so nobody has to catch them by hand.
Less wasted compute
Jobs are matched to nodes on CPU, memory and VRAM together, and a job that finishes, times out or is cancelled leaves nothing running behind it.
No duplicated or lost work
Each job has one owner, enforced by a lease. If the owner disappears, the job returns to a healthy worker without a person stepping in.
Clear to operate
A dashboard shows nodes, jobs, the event history of each job and a live cluster event feed, so an operator can see what happened and why.
One platform for many customers
Tenant isolation is enforced in the application and in the database, and the tenant is taken from the credential, never from the request.
Upgrades you can trust
Each release is rehearsed against a database from the release before it, so an upgrade is something you have already watched work.
Yours to run and inspect
Open source under the MIT licence and started with Docker Compose. Every route needs an API key, and the reasoning behind each design choice is public in the decision records.
Built for the day something breaks
Machines fail, processes linger and releases get rolled out. AetherGrid is designed around those moments.
A worker disappears
Its lease runs out, reconciliation finds the abandoned job, and the job goes back to a healthy worker.
Two workers want the same job
Only the worker holding the lease may act on it, and a worker that loses its lease stops acting as the owner.
A job leaves processes behind
When a job ends, times out or is cancelled, its whole process tree is stopped, not only the first process.
The scheduler runs in two places
The cluster tick runs in one process at a time, so two schedulers never place the same work.
Customers share one database
Tenant isolation is enforced in the application and in the database, so a worker, a job or a lease cannot be tied to another tenant’s records.
You upgrade to a new release
Each release is rehearsed on a database from the one before it: migrate, compare every table and the schema, then start the new API and read the data back.
Technical depth
The guarantees above rest on specific engineering decisions. Each one is written down in a decision record with the reason it was made.
Lease fencing
Release, renewal and outcome reports are fenced by lease identity, not only by worker identity, so a stale worker cannot change a job it no longer owns.
Reconciliation
Bounded-retry reconciliation reclaims work from dead workers, expired leases and offline nodes through one shared reclaim operation.
One scheduler at a time
Scheduler leadership is elected with a PostgreSQL advisory lock, so two scheduler processes never place the same work.
Process supervision
Job commands start under a supervisor that stops the whole process group when the owner dies. Timeouts end with SIGTERM, then SIGKILL.
Tenant boundary in the database
On PostgreSQL, composite keys tie workers and leases to their tenant, and the migration stops with a count if any row crosses tenants or has none.
Layers that are enforced
Domain, Application, Infrastructure and Presentation stay separate, with a test that fails if an import crosses a layer boundary.
Engagements
Three ways to work together. Each one starts with a technical consultation and a written scope.
Reliability review
A focused look at where your AI workloads can be lost, duplicated or stranded, with clear recommendations your team can act on.
Architecture design
Execution ownership, recovery and observability designed for your distributed compute, with the reasoning written down.
Pilot deployment
AetherGrid running against your workloads, with a defined scope, so you see the results on your own infrastructure.
Every claim has a link
The code, the release history and the reasoning behind each design choice are public. Read them, run them, or try the dashboard.
Technical stack
Languages
- Python
- Go
- TypeScript
- JavaScript
- SQL
Backend
- FastAPI
- REST API design
- Async Python
- Pydantic
- PostgreSQL
- Node.js
Distributed systems
- Scheduling
- Leases
- Reconciliation
- Concurrency
- Retries with backoff and jitter
- Circuit breakers
- Idempotency
- Worker heartbeats
- Kafka
Cloud and DevOps
- AWS
- ECS
- Kubernetes
- Terraform
- Docker
- GitHub Actions
- CI/CD
Security
- API key authentication
- Scoped authorization
- Tenant isolation
- Container hardening
Frontend
- React
- TypeScript
- Vite
- Tailwind CSS
- Playwright
Design and testing
- Clean Architecture
- DDD
- ADRs
- TDD
- pytest
- Contract testing
- Integration testing
- End-to-end testing
Wycliffe Kipleting Rotich
Platform engineer and creator of AetherGrid. I help teams building AI infrastructure make their workloads reliable, with clear ownership, safe recovery and systems their operators can understand. Based in Nairobi, working with teams remotely.
I build and review the infrastructure underneath AI workloads: scheduling, execution ownership, failure recovery and observability. I created and maintain AetherGrid, an open-source control plane for AI compute, and I bring the same discipline to the systems of the teams I work with.
The experience behind the work
Founder and Platform Engineer, AetherGrid
Designed and built AetherGrid end to end with Python, FastAPI, PostgreSQL, React, TypeScript, Docker Compose and GitHub Actions. A best-fit scheduler across CPU, memory and VRAM, time-bound leases, and reconciliation that reclaims work from dead workers, expired leases and offline nodes. Race conditions closed by fencing on lease identity, a subprocess engine with enforced timeouts and cancellation, key authentication on every route, hardened containers, and tenant scoping where the tenant comes from the credential, never the request.
LLM Technical Trainer, Mercor
Evaluated LLM-generated code for correctness, architecture, concurrency and API behavior, and wrote technical feedback that improves model reasoning and code quality.
AI Data Specialist, Atlas Capture
Annotated and reviewed video datasets for machine-learning workflows, and resolved inconsistent or ambiguous annotations in QA reviews.
Software Engineer, Cloud Platforms, Toyota
Fixed race conditions in Kafka-based ingestion pipelines for vehicle and telematics event data, and deployed containerized workloads on Kubernetes and ECS across AWS, with infrastructure provisioned in Terraform.
Backend Cloud Engineer, Kenya Revenue Authority
Worked on production revenue systems handling real financial data at national scale, and traced race conditions across application logic, shared state and database interactions.
Education
MSc Computer Science
Jan 2022 to Dec 2024
BEng Computer Software Engineering
Aug 2014 to Nov 2018
Frequently asked questions
What is AetherGrid?
An open-source control plane that keeps AI workloads running when machines fail. It schedules jobs, gives each one a clear owner and recovers work automatically.
What happens when a machine fails mid-job?
The system notices and recovers. Every job is owned through a renewable lease, and when a worker stops renewing it, reconciliation returns the job to a healthy worker. Nobody has to be paged for the routine case.
How do you prevent duplicated work?
Ownership is explicit. Each job has one authorized owner at a time, and recovery is designed so a reclaimed job is not left with two owners.
Can we see it working before we commit?
Yes. A live dashboard is available today and the full source is public. A scoped pilot lets you see it on your own workloads.
Can we inspect the design and the code?
Completely. The source, releases and written architecture decisions are open, so your engineers can verify every claim for themselves.
How is access controlled between teams?
Scoped API keys and tenant isolation keep each team’s jobs, nodes and workers separate.
Will it fit our existing stack?
Yes. AetherGrid focuses on execution ownership and recovery, so it is designed to work alongside tools like Kubernetes, Ray and Temporal rather than replace them.
What does an engagement look like?
Three clear options: a reliability review, an architecture design or a scoped pilot. Each starts with a technical consultation and ends with a written scope.
Who is behind it?
Wycliffe Kipleting Rotich, a platform engineer in Nairobi who built AetherGrid and has worked on production systems at Toyota and the Kenya Revenue Authority. Available for remote engagements.
How do we start?
Book a technical consultation. You leave with a clear summary of scope and next steps.
Planning AI infrastructure that has to keep running?
Tell me about your workloads and where they are fragile. I read every message.
Book a technical consultation