Secret rotation procedure

Audience: Platform operators (Plane-1 identities — see ADR 0006) ADO tracking: AB#3350 (Story), AB#3351 (KV policies), AB#3352 (this doc) Last updated: 2026-06-20

---

Overview

All production secrets are stored in the Azure Key Vault instance provisioned by infrastructure/modules/keyvault.bicep. No secret is committed to source control, environment files, or container image layers.

The Container App API service reads secrets at runtime via its system-assigned managed identity (Key Vault Secrets User role on the vault). The managed identity is granted by keyvault.bicep — no manual access policy management is required.

---

Secret inventory

Secret name (KV)	Rotated every	Owner	What uses it
database-url	90 days	Platform operator	apps/api — DATABASE_URL env var
clerk-secret-key	90 days	Platform operator	apps/api — CLERK_SECRET_KEY env var
child-jwt-secret	90 days	Platform operator	apps/api — CHILD_JWT_SECRET env var
twilio-auth-token	180 days	Platform operator	apps/api — TWILIO_AUTH_TOKEN env var
sendgrid-api-key	180 days	Platform operator	apps/api — SENDGRID_API_KEY env var
postgres-admin-password	180 days	Platform operator	PostgreSQL admin access only

---

Rotation procedure

1. Database URL (database-url)

1. Connect to PostgreSQL as the admin user.
2. Create a new application user with a strong password:

   CREATE USER hch_app_v2 WITH PASSWORD '<new-password>';
   GRANT CONNECT ON DATABASE hch TO hch_app_v2;
   GRANT USAGE ON SCHEMA public TO hch_app_v2;
   GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO hch_app_v2;
   GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO hch_app_v2;

3. Update the Key Vault secret:

   az keyvault secret set `
     --vault-name <kv-name> `
     --name database-url `
     --value "postgresql://hch_app_v2:<new-password>@<host>:5432/hch?sslmode=require"

4. Restart the Container App to pick up the new value (or wait for the next revision):

   az containerapp revision restart --name <app-name> --resource-group <rg>

5. Verify the API health endpoint returns 200 and application logs show no auth errors.
6. Drop the old user after confirming successful rotation:

   DROP USER hch_app;

2. Clerk secret key (clerk-secret-key)

1. Generate a new API key in the Clerk Dashboard under API Keys → Backend API keys → New key.
2. Update Key Vault:

   az keyvault secret set --vault-name <kv-name> --name clerk-secret-key --value "<new-key>"

3. Restart the Container App.
4. Verify authentication flows (sign-in, session exchange) succeed.
5. Revoke the old Clerk API key from the dashboard.

3. Child JWT secret (child-jwt-secret)

1. Generate a cryptographically random 256-bit value:

   [System.Convert]::ToBase64String([System.Security.Cryptography.RandomNumberGenerator]::GetBytes(32))

2. Update Key Vault:

   az keyvault secret set --vault-name <kv-name> --name child-jwt-secret --value "<new-value>"

3. Restart the Container App.
4. Warning: All existing child session tokens (platform JWTs) are immediately invalidated. Children will need to sign in again. Notify parent members before rotating in business hours.

4. Twilio auth token (twilio-auth-token)

1. Roll the auth token in the Twilio Console under Account → Auth Tokens → Secondary → Promote.
2. Update Key Vault:

   az keyvault secret set --vault-name <kv-name> --name twilio-auth-token --value "<new-token>"

3. Restart the Container App and confirm SMS delivery.

5. SendGrid API key (sendgrid-api-key)

1. Create a new API key in SendGrid under Settings → API Keys → Create API Key with Restricted Access (Mail Send only).
2. Update Key Vault:

   az keyvault secret set --vault-name <kv-name> --name sendgrid-api-key --value "<new-key>"

3. Restart the Container App and send a test email via the admin panel.
4. Delete the old API key in SendGrid.

---

Per-identity access policies (AB#3351)

Only the following identities hold access to the Key Vault:

Identity	Role	Granted by
Container App system-assigned MI	Key Vault Secrets User (4633458b-…)	keyvault.bicep (automated)
GitHub Actions OIDC federated credential	Key Vault Secrets User	Manual grant (run once at platform setup)
Platform operators (Kristopher Turner + backup)	Key Vault Secrets Officer	Manual grant via az role assignment create

No community member, minister, or application user holds any Key Vault access. The OIDC federated credential grants CI/CD pipelines read access for deploying secrets as Bicep parameters during the initial bootstrap only; after that, secrets are rotated manually by operators.

Grant GitHub Actions access (one-time setup)

# Get the OIDC app's object ID
$oauthObjectId = az ad app show --id <github-app-client-id> --query objectId -o tsv

az role assignment create `
  --role "Key Vault Secrets User" `
  --assignee-object-id $oauthObjectId `
  --assignee-principal-type ServicePrincipal `
  --scope "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<kv-name>"

---

Alerts

An Azure Monitor alert rule (defined in observability.bicep) fires when:

• A Key Vault secret version is accessed more than 500 times in one hour (anomaly)
• Any IAM change is made to the Key Vault (audit)

These alerts route to the platform operations email configured in the alert action group.