Database reliability
Build recovery into every write path. A successful Database response confirms the requested data operation; it does not establish a transaction across multiple HTTP requests, schema metadata, event delivery, and downstream consumers.
What succeeds together
| Scope | Behavior |
|---|---|
| One record-write request | Database executes the write in one transaction |
| One bulk create, update, or ID-delete request | Database executes the records in that request in one transaction |
| Several HTTP requests | Each request is independent |
| Schema definition and active table structure | Updated in separate transactions; verify both after a change |
| Record write and event delivery | The saved record is the source of truth; consumers must handle delayed or missing notifications |
Put the writes that must succeed together in one request. For work spanning several requests, save your progress after each step and make every step safe to repeat.
Use stable record identifiers
Keep the generated ID from every create response. When a workflow needs a stable identity before the request begins, supply an application-owned UUID. Either approach lets the application read the target state after an interrupted request instead of immediately creating a duplicate.
The Database API does not define an idempotency-key header, entity
version, ETag, or If-Match precondition for record writes.
Retry by operation
| Situation | Recovery action |
|---|---|
| List or direct read interrupted | Retry with the same bounded query |
| Create response interrupted | Read by the application-owned ID when one was supplied; otherwise reconcile using the operation's stable business key before retrying |
| Update response interrupted | Read the record and compare the intended fields before deciding whether to repeat the update |
| Delete response interrupted | Read by ID; treat an absent record as the desired end state |
| Bulk write interrupted | Read every target ID and reconcile each record before sending another batch |
| Schema operation interrupted | Stop application writes; verify the stored descriptor and physical table structure before another change |
Use bounded exponential backoff for transient connection failures. Do not blindly retry validation, authorization, conflict, or schema errors.
Plan for concurrent writes
Flat-table record updates do not expose row versions or optimistic concurrency checks. Two clients that read the same record and then write derived values can overwrite one another; the last completed write can win.
For counters, balances, inventory, or state transitions that must not lose an update, perform the change in one server-side step, such as a function, instead of reading, changing, and writing from the client.
Schema-definition changes are serialized. This does not create an application-facing concurrency contract for record writes.
Understand batch scope
Bulk endpoints reduce network round trips, but they are not a transaction across separate requests. Keep each batch small enough to:
- read back and reconcile;
- retry individual target IDs;
- fit within the query timeout; and
- observe failures without replaying unrelated work.
Do not combine a schema change and record migration into one assumed atomic application operation.
Treat schema changes as migrations
The schema definition and active table structure change through separate stages. A failed change therefore needs explicit verification and recovery; a single application request does not establish an atomic rollback across both stages.
Retain an external recovery point, apply one reviewed schema change at a time, and verify the stored descriptor, physical schema, and table behavior before releasing dependent code. See Plan schema changes.
Design event consumers for recovery
A successful mutation response confirms the Database write; it is not a receipt for downstream event processing. Treat event delivery as a separate boundary and make every consumer able to recover from delays, duplicates, or missing notifications.
Consumers should:
- Treat an event as a notification, not the only copy of state.
- Make event processing idempotent using the event or record identity.
- Read current Database state when exact state matters.
- Reconcile missed work from a durable application checkpoint.
- Reconcile from a durable checkpoint instead of assuming one notification exists for every mutation.
Operational checklist
- Keep generated IDs from create responses, or reuse application-owned UUIDs when a retry must preserve identity.
- Bound every list and batch.
- Read state after ambiguous write failures.
- Avoid client-side read-modify-write for critical invariants.
- Keep schema and record migrations observable and recoverable.
- Make downstream consumers idempotent.
- Record the status, time, and operation of each request, without credentials.
Continue with: