Operations · Hosted environments
One control plane provisions, names, renews and reclaims every tenant environment
The hosted-environments platform as it runs: a browser surface, one control plane and one private provisioner in a pinned Fly organization, two shared storage parents, and the per-tenant environments, shared data tiers, real hostnames and credentials they produce. Everything here is observed on the live deployment between 2026-09-13 and 2026-10-05; the dated narrative and every finding are in docs/delegation/deploy-and-renewal-results-2026-09-18.md.
Operations · Hosted environments
One control plane provisions, names, renews and reclaims every tenant environment
The hosted-environments platform as it runs: a browser surface, one control plane and one private provisioner in a pinned Fly organization, two shared storage parents, and the per-tenant environments, shared data tiers, real hostnames and credentials they produce. Everything here is observed on the live deployment between 2026-09-13 and 2026-10-05; the dated narrative and every finding are in docs/delegation/deploy-and-renewal-results-2026-09-18.md.
What this view establishes
The architectural commitments
- The hosted platform is one live control plane and one private provisioner in a pinned Fly organization, operated from a browser: every tenant environment is planned, created, duplicated, restored, retired, abandoned and reclaimed through routes that validate their own contracts in both directions.
- Authority is short-lived and the deployment renews its own: a root credential on the control volume fathers every 168-hour lease and parent token by attenuation, an hourly tick rotates each due tenant in expiry order, and the laptop keeps one quarterly mint and an operator's resume of the two fence states a deployment will not resume for itself.
- A shared data tier is the only place two environments share data, and it says so: members of one tier share its database, approval list, identity table and token table, while a member owns no storage, carries no backup and can hold no public name.
- Reclamation is not recovery. Nothing re-enters a failed dispatch: five verbs fence the journals, revoke routing first, observe every provider reference and tombstone only when all are absent -- and a durable retire keeps its data for the owner's seven days before the sweep takes it.
Layered platform stack
Operating surface
Deployment console
One screen: environments grouped by project and domain, an authority fold, an admit-project form, a needs-attention banner, shared data tiers per project, and a detail aside whose buttons plan, create, duplicate, restore into a new environment, retire, abandon and expire a backup. It accepts no browser bearer and derives no URL; every link comes from the validated projection.
The Explorer's proxy
The hosted web role is this repository's Explorer image. It serves /deployments and proxies an exact allow-list of control-plane routes as the signed-in session, validating each against the contract version that route answers. Validating the wrong version is a 503 the moment control answers a newer one (finding 69).
The deployment itself
The control plane
One Machine in the managed Fly organization, on a durable volume, serving about twenty-five routes in eight families: environments, operations, records-v2, tenants, releases, admissions, authority and tiers. It holds the role store, the provider journal, the parent token map, the root credential and the zone token. Recovery is not routed.
Validated wire contracts
Request bodies and emitted projections are checked both ways. deployment-control-v1 is frozen; the hosted projection grows one additive version at a time (v9 carries tiers), and operator, authority v2, reclamation, duplicate and tenant-admin each declare their own refusal words so a refusal a provider produced never reaches a browser as a sentence.
The private provisioner
A second Machine with no public listener, UI or MCP route, reached over a mutually authenticated private channel. It holds the short-lived organization lease, re-derives every specification it is handed, and mints each child credential by attenuation. An image that cannot decode a specification shape says so: provisionerSpecUnrecognized, 409.
Four sweeps in the service
Daemon threads inside the process that holds the journal, each with an interval from the composition: reclamation (hourly), hostname confirmation (every minute), the certificate pass (every six hours) and renewal (hourly). A receipt is written only when a tick did work or went wrong, and nothing raised escapes a tick.
The deployment renews itself
Hosted authority is short-lived by design: a 168-hour window, due at half of it. A long-lived root credential on the control volume fathers every lease and parent token by client-side attenuation; control writes its own files in place and restarts only the provisioner. The laptop keeps the quarterly mint and an operator's resume.
Reclamation and retention
Five verbs let an allocation go without ever retrying it: reclaim, reclaimFailedDelete, reclaimUnjournalledDelete, reclaimUnjournalledCreate and expireRetained. Each fences the journals, revokes routing first, observes every reference and tombstones only on all-absent. A durable retire keeps its database, bucket and backup prefix for seven days.
What a tenant gets
Two shared parents
One PostgreSQL Machine and one MinIO Machine per deployment, pinned to exact manifests that are mirrored into the deployment's own registry, because a parent whose image exists only upstream is a parent somebody else can delete. No parent is ever deleted; a parent is observed by shape and accepted on its pin or an operator-named successor.
A child environment
An immutable server-owned specification binds environment, authority, tenant, stage, parent ownership, release and image digests and six exact resources, seven where a deployment carries an identity provider. Its gateway and web Machines are its own; its database, role, bucket, user and policy are named from its uuid in derivations the tests hold against each other.
A shared data tier
The one deliberate exception to one-environment-one-database: a named, durable database and bucket under a project that admitted members attach to, with its own journal rows and four resources. A member brings its own login and object key and owns no storage; members share the approval list, the identity table and the token table. A tier retires only with no member left. Built but not yet deployed (applications charter part 2c): a tier made on a deployment holding the vault and a worker adapter also carries a fifth resource, its worker Machine (ns-t-<hex>-wk, no services, its token a vault-held Fly secret whose digest members carry), allocated on the first member that can serve it, pointed at the newest, stopped with none, destroyed first on retire; the four-kind tuple stays valid for every earlier tier.
Real hostnames and certificates
A durable environment can answer on two bare hosts under one zone: DNS-only CNAMEs to its Fly names, an _acme-challenge CNAME written to the target the certificate object states, and Fly certificates issued and removed by the control plane over the provider's GraphQL API. A confirmed name's certificates are observed on a slower pass and alerted.
Sign-in, per child
One hosted account authority is carried into every child: the control plane owns one Auth0 application's callback list and adds exactly one URL on create and removes exactly one on retire, as a journalled resource kind of its own. Identity is shared; data is per deployment.
The binding a child boots on
A release carries a runtime closure and an application manifest; a child receives a closed binding whose keys the gateway's own closure must admit (access_policy, hostname, restore, tier, bootstrap_admin). Three separately shipped key sets must agree, so a new key costs a gateway image and a release -- findings 62 and 73 are each one of those. From bootstrap_admin on, a release declares the optional keys its image admits (childSpecExtensions) and control emits one only to a release that declares it.
Connections and evidence
Open the 16-relationship source key
Numbers set an explanatory reading order. They do not measure runtime timing.
Connections and evidence
Open the 16-relationship source key
- 01observed
Deployment console→The Explorer's proxy
same-origin, no browser bearer
- 02observed
The Explorer's proxy→The control plane
allow-listed routes as the session
- 03declared
The control plane→Validated wire contracts
every body and projection validated
- 04observed
The control plane→The private provisioner
registers a child privately
- 05observed
The control plane→Four sweeps in the service
ticks from its own composition
- 06observed
Four sweeps in the service→The deployment renews itself
hourly: is any tenant due
- 07observed
Four sweeps in the service→Reclamation and retention
due deletes, expired copies, retained tiers
- 08observed
The deployment renews itself→The private provisioner
attenuated lease, one Machine update
- 09observed
The control plane→A child environment
journalled steps, exact resources
- 10observed
A child environment→Two shared parents
its own database, bucket and policy
- 11observed
A shared data tier→Two shared parents
the tier owns the database and bucket
- 12observed
A child environment→A shared data tier
a member binds instead of provisioning
- 13observed
The control plane→Real hostnames and certificates
a CNAME, then a certificate
- 14observed
The control plane→Sign-in, per child
one callback added, one removed
- 15implemented
A child environment→The binding a child boots on
a composition read once at boot
- 16observed
Reclamation and retention→A child environment
observed absent, then tombstoned
Connections and evidence
16 directed, source-backed relationships
Numbers set an explanatory reading order. They do not measure runtime timing.
- 01observed
Deployment console→The Explorer's proxy
same-origin, no browser bearer
- 02observed
The Explorer's proxy→The control plane
allow-listed routes as the session
- 03declared
The control plane→Validated wire contracts
every body and projection validated
- 04observed
The control plane→The private provisioner
registers a child privately
- 05observed
The control plane→Four sweeps in the service
ticks from its own composition
- 06observed
Four sweeps in the service→The deployment renews itself
hourly: is any tenant due
- 07observed
Four sweeps in the service→Reclamation and retention
due deletes, expired copies, retained tiers
- 08observed
The deployment renews itself→The private provisioner
attenuated lease, one Machine update
- 09observed
The control plane→A child environment
journalled steps, exact resources
- 10observed
A child environment→Two shared parents
its own database, bucket and policy
- 11observed
A shared data tier→Two shared parents
the tier owns the database and bucket
- 12observed
A child environment→A shared data tier
a member binds instead of provisioning
- 13observed
The control plane→Real hostnames and certificates
a CNAME, then a certificate
- 14observed
The control plane→Sign-in, per child
one callback added, one removed
- 15implemented
A child environment→The binding a child boots on
a composition read once at boot
- 16observed
Reclamation and retention→A child environment
observed absent, then tombstoned
What this view establishes
The architectural commitments
- The hosted platform is one live control plane and one private provisioner in a pinned Fly organization, operated from a browser: every tenant environment is planned, created, duplicated, restored, retired, abandoned and reclaimed through routes that validate their own contracts in both directions.
- Authority is short-lived and the deployment renews its own: a root credential on the control volume fathers every 168-hour lease and parent token by attenuation, an hourly tick rotates each due tenant in expiry order, and the laptop keeps one quarterly mint and an operator's resume of the two fence states a deployment will not resume for itself.
- A shared data tier is the only place two environments share data, and it says so: members of one tier share its database, approval list, identity table and token table, while a member owns no storage, carries no backup and can hold no public name.
- Reclamation is not recovery. Nothing re-enters a failed dispatch: five verbs fence the journals, revoke routing first, observe every provider reference and tombstone only when all are absent -- and a durable retire keeps its data for the owner's seven days before the sweep takes it.
Follow the live system
Move from explanation to inspection
The Atlas is static and source-backed. The Observatory shows authenticated environment state.
48 repository sources behind this view
Explorer/README.mdExplorer/app/lib/deploymentControl.server.tsdocs/delegation/durable-reclamation-charter-2026-09-29.mddocs/delegation/renewal-on-the-deployment-charter-2026-10-02.mddocs/delegation/shared-runtime-tiers-charter-2026-10-05.mdplatform/contracts/deployment-control-authority-v2.jsonplatform/contracts/deployment-control-hosted-read-v9.jsonplatform/contracts/deployment-control-operator-v1.jsonplatform/contracts/deployment-control-reclamation-v1.jsonplatform/contracts/deployment-control-tenant-admin-v1.jsonplatform/contracts/deployment-control-v1.jsonplatform/control-plane/README.mdplatform/control-plane/control_plane.pyplatform/control-plane/hosted-child-spec-limitations.mdplatform/control-plane/hosted_alerts.pyplatform/control-plane/hosted_allocation_reaper.pyplatform/control-plane/hosted_auth0_management.pyplatform/control-plane/hosted_authority_window.pyplatform/control-plane/hosted_certificates.pyplatform/control-plane/hosted_child_provider.pyplatform/control-plane/hosted_child_spec.pyplatform/control-plane/hosted_cloudflare_dns.pyplatform/control-plane/hosted_control_rotation.pyplatform/control-plane/hosted_hostname_effects.pyplatform/control-plane/hosted_identity_callbacks.pyplatform/control-plane/hosted_identity_provider.pyplatform/control-plane/hosted_lifecycle_service.pyplatform/control-plane/hosted_operator_service.pyplatform/control-plane/hosted_parent_observer.pyplatform/control-plane/hosted_provisioner_credential.pyplatform/control-plane/hosted_provisioner_rpc.pyplatform/control-plane/hosted_provisioner_service.pyplatform/control-plane/hosted_restore_effects.pyplatform/control-plane/hosted_root_attenuation.pyplatform/control-plane/hosted_root_credential.pyplatform/control-plane/hosted_storage_effects.pyplatform/control-plane/hosted_tier_effects.pyplatform/control-plane/hosted_tier_journal.pyplatform/control-plane/hosted_tier_provider.pyplatform/control-plane/hosted_tier_spec.pyplatform/deployment-console/README.mdplatform/deployment-console/src/app.jsplatform/deployment-console/src/client.jsplatform/hosted-pilot/lifecycle/README.mdplatform/hosted-pilot/lifecycle/disposable_parents.pyplatform/hosted-pilot/runtime-closures/application-manifest.jsonplatform/hosted-pilot/runtime-closures/build_context.pyplatform/hosted-pilot/runtime-closures/hosted_child_binding.py