[prasadlohakpure]

AWOL-on-rollout plan

This PR is Phases 1 and 2.

• Phase 1 (this PR): make SIGTERM reach Heimdall (exec in entrypoint; ECS stopTimeout 120s in deploy repo)
• Phase 2 (this PR): drain on shutdown (stop claiming, HTTP shutdown, wait for in-flight jobs and log archiving)
• Phase 3: reattach instead of fail (plugin.Resumer for sparkeks and ecs)
• Phase 4 (optional): unpicked_job_timeout for jobs never claimed, off by default

Problem

During rollouts, async jobs running on the old task are failed with awol: job is stale, marking failed. The janitor's plugin Cleanup then deletes their Spark apps or stops their ECS tasks, which were usually still healthy.

Heimdall never saw the container's SIGTERM, and had no shutdown handling anyway. entrypoint.sh started it in a subshell, so bash was PID 1. ECS sends SIGTERM to PID 1, waits stopTimeout, then SIGKILLs everything. Every in-flight job, sync request and log archive copy died at once.

Changes

Phase 1: entrypoint.sh

Runs deploydb and then exec ./heimdall at the top level, so Heimdall replaces the shell as PID 1 and receives SIGTERM directly. Next.js still starts in the background as before.

Phase 2: graceful drain

On the first SIGTERM or interrupt, each step waiting up to shutdown_timeout seconds. The default comes from the HEIMDALL_SHUTDOWN_TIMEOUT environment variable, which forklift sets from stop_timeout (patterninc/forklift#152); without it, 30 seconds, the ECS default. A shutdown_timeout in the config file overrides both:

1. The pool feeder stops claiming. Unclaimed jobs stay new, and another task picks them up.
2. http.Server.Shutdown: new connections are refused, and in-flight sync requests finish.
3. The pool waits for claimed async jobs to finish and write their final status.

A second signal exits immediately. If a step times out, Heimdall exits non-zero with context deadline exceeded. Jobs still running at that point are handled by the janitor exactly as today; Phase 3 addresses them.

Impact

• Only the shutdown path changes. Rollouts get slower by up to the drain window per task.
• Not waited for: the goroutines that run after a job or request returns (log archive copy, sync insertJob, sync storeResults). They're usually done within the drain, but one still running at exit is cut off as today.
• Next.js is re-parented to Heimdall instead of bash. Go doesn't reap zombie processes; that only matters if Next.js exits, and it runs for the life of the container.
• If deploydb fails, the container still exits without starting Heimdall, as before.

Follow-up outside this repo

Set stop_timeout: 120 (the Fargate maximum) in the deploy repo's forklift.yaml: patterninc/heimdall-config#549. Forklift passes the same value in as HEIMDALL_SHUTDOWN_TIMEOUT, so the drain always matches the real stop timeout.

Testing

Immediate escape when no pending jobs are there (immediately exiting)

Waiting for active jobs to drain (waiting for longer period)

[Copilot]

🟢 Approval recommended

No active review issues are identified for the entrypoint signal-handling change.

No open issues 🎉

What changed in this PR

Implements Phase 1 of the AWOL-on-rollout plan by letting Heimdall receive container SIGTERM directly. Graceful draining remains deferred to Phase 2.

Changes:

• Runs database deployment and Heimdall startup at the top level, using exec to replace the entrypoint shell.
• Preserves background Next.js startup and prevents Heimdall from starting if database deployment fails.

File	Description
entrypoint.sh	Removes the Heimdall startup subshell and uses exec so Heimdall becomes PID 1.

_{Review effort: Balanced}