Support/Operations runbooks/Core service crash-loop response
/

Operations runbooks

Core service crash-loop response

How BuildWithHQ distinguishes controlled deployments from repeated automatic core-service restarts and routes the alert safely.

Warning
2 automatic restarts in 15 minutes
High
4 automatic restarts in 15 minutes
Tenant workloads
Monitored separately

BuildWithHQ monitors automatic systemd restarts for its API, authentication, control-plane, tenant-runtime, proxy, Harness, and release-bound worker services. The alert carries the exact service and unit instance needed by an operator, without tenant, user, database, credential, or correlation identifiers.

What triggers the alert

The signal uses systemd's automatic NRestarts counter. A controlled deployment that stops one release and starts another does not increment that counter. Two automatic restarts within 15 minutes create a warning ticket; four or more create a high-severity page and ticket. The high condition replaces the warning condition for the same unit.

Tenant Firecracker workloads use a separate container-fleet signal and recovery path. A tenant workload identity cannot enter the core-service metric.

First response

  1. Confirm the core-systemd exporter is fresh. Missing telemetry is treated as a monitoring failure, not as proof that the service is healthy.
  2. Use the alert's service, exact unit, unit instance, host scrape instance, and time window to inspect only the affected service.
  3. Confirm the unit is running the expected signed release and that its dependencies and readiness endpoint are healthy.
  4. Review secret-scrubbed service logs and authenticated traces for the alert interval. Correlation IDs stay in the protected trace system.
  5. Use the health-gated deployment or worker lifecycle to roll forward or back. Do not edit live executables, environment files, leases, or database state to silence an alert.

When the incident is resolved

Resolve the alert only after the exact unit remains stable for a complete 15-minute window, its readiness and dependent workflow checks pass, monitoring output is fresh, and both warning and high conditions clear. Public incident text should describe only the affected platform capability and recovery status; exact hosts, units, restart counts, correlations, and internal diagnostics remain private.

Capability review: 2026-09-14. For exact current technical availability, use the generated API Map and first-class module inventory.