A domain that stands on its own
Business rules sit in plain Python and do not depend on the web framework or the database. The layer boundaries are enforced by a test, so the structure stays as designed as the system grows.
Architecture, built to be read
AetherGrid is organised in four layers with dependencies pointing inward. The domain holds the rules, the outer layers hold the details, and every non-obvious decision is written down in a public decision record.
Presentation
FastAPI endpoints for jobs, nodes, workers, events, cluster health and API keys. They validate input, call an application service and return the response. No business logic lives here.
Application
Services that coordinate domain objects and repositories. The worker execution loop runs each job as a real subprocess and keeps its lease renewed. The reconciliation loop recovers crashed workers, expired leases and inconsistent state.
Domain
Job, Node, Worker, Lease, Event and ApiKey enforce their own invariants. Scheduling policy and the job lifecycle live here as plain Python, with no FastAPI or psycopg imports.
Infrastructure
PostgreSQL repositories written with raw psycopg, so query behaviour and transaction boundaries stay visible. Every repository is validated by a shared contract test suite.
How recovery stays safe
Ownership through leases
A job is owned through a renewable lease. The worker renews it continuously for the whole run, and if a renewal fails the worker discards its result instead of writing it against another owner's work.
Reconciliation as a core mechanism
A reconciliation loop finds crashed workers, expired leases and jobs abandoned mid-cancellation, and repairs them. It re-checks lease validity at the moment it acts.
One scheduler, elected safely
A single scheduler is elected with a database advisory lock, and the cluster tick runs off the event loop so the API stays responsive while the cluster is being managed.
Persistence you can reason about
Raw psycopg keeps every query and transaction boundary explicit. PostgreSQL and the local backends are held to the same behaviour by shared contract tests, and the schema moves forward through versioned migrations.
Workers that clean up after themselves
A job's command is stopped along with anything it left running, whether the job finishes, times out, is cancelled or loses its owner. A command that cannot start fails with a clear outcome.
Want a walkthrough of the design?
Book a technical consultation and we will go through it together.
Book a technical consultation