Operations and troubleshooting

The current isolated-target procedures and test evidence are in the v14 runbook. Disposable encrypted restore drills passed for MySQL/MariaDB/PostgreSQL; live production recovery remains unverified.

Production origin: https://asi-database-node-api.arybit.co.ke

Check /health, /ready and /version separately. A healthy process can have an unavailable database. Verify the expected revision after release promotion. On the VM, inspect the selected systemd service and Nginx configuration; deployment service names depend on release state.

bash run.sh --13 --verify
bash run.sh --13 --logs --stream

Private runtime files and backup artifacts live under /opt/asi-database-node-api/shared/ on the managed VM. Do not copy secrets or log contents into public documentation.

For HTTPS failures, check domain/alias DNS, certificate coverage, expiry and Nginx configuration. For 400 responses check ALLOWED_HOSTS. For browser cross-origin failures check exact HTTPS CORS origins. For unavailable console operations, check database-node-setup.service, the operator token, and the prerequisites listed in console operations. The old cookie-session adapter is no longer used by the console.

Backups use a separate protected database identity and age encryption. Configure backup_recipient and backup_env_file; retain the decryption identity off-host. A checksum and encrypted archive are not proof of recoverability. Use python deploy-database-api/ops/restore-backup.py --help to prepare an isolated restore, then verify row counts, tenant roles, grants and application behavior. Release rollback does not restore database state.

If deployment is interrupted, inspect protected logs and the recorded state before using resume. Do not erase state or rerun migrations to mask an unresolved failure.

Direct database and HTTPS operation

Use the dual-interface runbook and public network proposal. API restarts do not restart the database; closing the public 3306 NSG rule does not stop HTTPS. A database restart/outage affects both. Do not fail over silently between interfaces after an uncertain write; reconcile the authoritative record first. Rotate direct DB identities and HTTPS credentials independently. Keep routine definers stable and review mapping expiry/revocation alongside DB account locks.

Backup client option files now require verified TLS for remote MySQL connections and choose the installed MySQL/MariaDB client's certificate-verification syntax. Use separate backup grants sufficient for data/routines/triggers. A restore of the new schema must include procedures, triggers, direct mappings and the routine definer/grant plan; do not blindly restore DEFINER clauses with an underprivileged account or grant global administrator rights to the existing isolated restore user. Disposable new-schema restore drills are implemented and tested; a live v14 restore remains blocked. Qualify it in an isolated MySQL/MariaDB instance before public cutover, preserving source data and protected credentials. Encrypted backup success alone does not demonstrate recoverability.

Monitor database authentication failures, TLS failures, connection saturation, server errors, migration ledger and backups separately from HTTP request logs. HTTP quotas are coordinated by service/tenant in the database; direct connections have individual account concurrency caps and need engine/platform abuse protection. No new direct-protocol metrics exporter or production log-retention service was installed. Audit and journal access must remain operator-only.