Skip to content

[awf] CLI: add --diagnostic-logs flag to capture container logs on failure #2011

Closed

Description

@lpcox

Problem

When AWF containers fail to start (e.g., Squid crashes on startup in DinD environments), there are no diagnostic logs — docker logs output, container exit codes, and mount info are never captured. Debugging requires customers to manually add diagnostic steps to their workflows.

Context

Original report: github/gh-aw#25548

A customer on ARC runners with DinD sidecars hit a Squid container crash (exit code 1) where the Squid access logs were empty (Squid never started). Diagnosing the root cause required multiple back-and-forth rounds.

Root Cause

The current cleanup lifecycle in src/cli.ts and src/docker-manager.ts calls docker compose down -v and removes the work directory, discarding all docker logs output and container state. There is no mechanism to capture container-level operational logs (stdout/stderr from entrypoints) before teardown.

The --keep-containers flag preserves containers but requires the user to know in advance they need it, and doesn't help for automated CI failures.

Proposed Solution

Add a --diagnostic-logs flag (off by default) to src/cli.ts that, when enabled and AWF exits with a non-zero code, collects container-level diagnostics before teardown:

  1. In src/cli.ts: Add --diagnostic-logs option to the Commander program definition.
  2. In src/docker-manager.ts: Add a collectDiagnostics(workDir, auditDir) function that runs:
    • docker logs awf-squid → diagnostics/squid-container.log
    • docker logs awf-agent → diagnostics/agent-container.log
    • docker logs awf-iptables-init → diagnostics/iptables-init.log (if exists)
    • docker inspect --format '\{\{.State.ExitCode}}' awf-squid awf-agent → diagnostics/exit-codes.txt
    • docker inspect --format '\{\{json .Mounts}}' awf-agent → diagnostics/agent-mounts.json
    • Sanitized docker-compose.yml (strip env var values containing tokens/keys) → diagnostics/docker-compose-sanitized.yml
  3. Call collectDiagnostics in the error/signal cleanup path (src/cli.ts:95-103, 122-126) when --diagnostic-logs is set and exit code is non-zero.
  4. Write to \$\{workDir}/diagnostics/ (preserved if --keep-containers, or moved alongside squid-logs).
  5. Security: Never collect raw env var values, full docker inspect JSON, or /etc/shadow; sanitize the compose file before writing.

Generated by Firewall Issue Dispatcher · ● 2.1M · ◷

Activity

  1. github-actions commented on Apr 16, 2026

    @github-actions
    Contributor

    👋 Hello! I noticed this issue might be related to existing issues:

    Both of these describe the same --diagnostic-logs feature for capturing Docker container logs on AWF failure, which closely matches this issue's proposal. If this work was already completed or the approach changed, it may be worth reviewing those issues.

    This is an automated message from the issue duplication detector.

    Generated by Issue Duplication Detector for issue #2011 · ● 235.1K · ◷

  2. lpcox commented on Apr 16, 2026

    @lpcox
    CollaboratorAuthor

    Fixed by #1906 and #1941 (--diagnostic-logs flag for container failure diagnostics). Merged to main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions