Runbook

Processes and ports

Process	Default bind	Public?
momo edge HTTP/HTTPS	0.0.0.0:80 / :443 (when enabled)	Yes (requires MOMO_ALLOW_PUBLIC_BIND=1)
momo gateway HTTP	127.0.0.1:8080	No (unless MOMO_ALLOW_PUBLIC_BIND=1)
momo node gRPC	127.0.0.1:50051	No
momo node metrics	127.0.0.1:9101	No
App containers	loopback dynamic publish	No

Local one-node start

momo node --node-id node-a --store file://./momo-state --auto-register
momo gateway --store file://./momo-state --bind 127.0.0.1:8080
momo apply -f examples/momo/app.yml --node node-a --store file://./momo-state --wait
curl -s http://127.0.0.1:8080/healthz

Detached equivalent (pid/logs under $MOMO_RUNTIME_DIR):

export MOMO_RUNTIME_DIR=./.momo-runtime
momo start all --store file://./momo-state --node-id node-a
momo ps
momo logs gateway -f

Production hosts should use systemd / momo service … with foreground momo gateway|node|edge — never momo start as ExecStart.

Common failures

Triage with doctor first:

momo doctor --store "$MOMO_STORE" --gateway-url http://127.0.0.1:8080

Symptom	Cause	Action
HTTP 404 unknown host	Domain missing from RouteSpec	momo routes; re-apply; check Host header
HTTP 503	Node unavailable / transport	momo nodes; check auth/TLS; journalctl -u momo-node
HTTP 504 / start timeout	Cold start or health gate	Node logs; docker ps -a; image health path; raise startup_timeout
Docker failures on reconcile	Daemon down / missing image	docker info; pull image; check MOMO_DOCKER_HOST
Gateway /readyz degraded	Route watch unhealthy or no nodes	Store connectivity; node heartbeats
Apply rejected	Spec / policy / placement	momo validate -f …; check drain / selectors
Production process refuses to start	Missing auth/TLS/JSON logs	Fill /etc/momo/momo.env; see Security
Doctor reports stale local detach	Leftover *.pid under runtime dir	momo stop --all or remove stale pidfiles
systemd unit exits immediately	ExecStart used momo start	Use foreground momo gateway|node|edge or momo service install
Edge ACME challenge 404	Token missing / HTTP not on edge	Ensure --http-listen reaches edge; check acme_issue_* logs; DNS A/AAAA
Edge ACME rate limit / issuance fail	Staging vs prod / retries	Use --acme-staging first; greplog acme_issue_failure; wait and retry
Edge TLS cert expired / clients fail	Renew window missed / fail-fast	Check acme_renew_* logs; verify acme-dir writable; restart after fixing DNS/:80
Edge upstream 502	Gateway down / unhealthy	curl gateway /healthz; check --upstream; edge health-check logs

ACME staging lab (manual)

Against Let’s Encrypt staging on a publicly reachable host:

1. Point DNS A/AAAA at the edge host; open TCP 80/443.
2. MOMO_ALLOW_PUBLIC_BIND=1 momo edge --acme-domain example.com --acme-email you@example.com --acme-staging --upstream 127.0.0.1:8080
3. Confirm greplog acme_issue_success and curl -vk https://example.com/healthz.
4. After renew/swap, new TLS handshakes present the new certificate without restarting the process.

Default CI never calls production Let’s Encrypt (make test / make edge-smoke are unit-only).

Backup and restore

Semantic backup (recommended)

momo export --store "$MOMO_STORE" -o "momo-backup-$(date +%F).yaml"
# restore into a fresh store
momo import -f momo-backup-YYYY-MM-DD.yaml --store "$MOMO_STORE_NEW"

After import, restart or wait for watches: gateway reloads routes; nodes reconcile HostSpecs; heartbeats resume when nodes are up.

Engine-level backup

Store	Method
Postgres	pg_dump / pg_restore of the Momo database plus periodic momo export
SQLite	Stop writers or use online backup; copy .db file; also keep momo export
File store	Copy the state directory atomically when idle; also keep momo export

Acceptance check after restore: momo routes / momo status match the export; gateway serves known Hosts once nodes reconnect; node reconciles HostSpec revision.

Rolling upgrade

1. momo export (safety snapshot).
2. Drain one node: momo node drain node-a.
3. Stop unit / process; install new momo binary; start node; confirm heartbeat.
4. momo node undrain node-a.
5. Repeat for other nodes.
6. Upgrade gateways last (or N−1 compatible — gateway is mostly forward-compatible).

Rollback: install previous binary; if schema/migration incompatible, restore Postgres/momo import from the pre-upgrade export. SQL migrations live under crates/momo-store/migrations/ — forward-only in production; rollback = restore DB backup + older binary.

Node maintenance

momo node drain node-a --store "$MOMO_STORE"
# wait for inflight; upgrade Docker / host
momo node undrain node-a --store "$MOMO_STORE"

Force unlink orphans AppSpecs — clean with remove / re-apply afterward.

Force cold start

docker stop <managed-container-name>
# next request Acquire cold-starts (single-flight under concurrency)