Activity

1. Kenny-Heitritter

   added

   wayfinder:researchWayfinder ticket: AFK researchWayfinder ticket: AFK research

   on Jul 24, 2026
2. Kenny-Heitritter

   added a parent issue on Jul 24, 2026

   • Wayfinder: CodeQ deployment — seamless staging/prod updates without stale binaries/configs #16
3. Kenny-Heitritter

   self-assigned this

   on Jul 24, 2026
4. Kenny-Heitritter commented on Jul 24, 2026

   

   Kenny-Heitritter

   on Jul 24, 2026

   MemberAuthor

   More actions

   Resolution — validated against source of truth

   Headline finding: CODEQ_DEPLOYMENT.md did not exist on any branch of
   qBraid/opencode (checked all local + remote branches and GitHub code search;
   blob/dev/CODEQ_DEPLOYMENT.md is a 404). So this ticket's "validate &
   complete" resolved to authoring the anchor from scratch against the live
   source of truth, not correcting an existing file's drift. Every factual claim
   carried in the map turned out to be accurate — the work was verification +
   adding precision, with no corrections needed.

   Verified against: qbraid-infrastructure main @ 01b278b; qbraid-Dstacks
origin/staging (bbf58c3) and origin/main (69999a9) — the branches that
   deploy to staging/prod. (Note: qbraid-Dstacks origin/staging's
   setup-codeq-config.sh has 6 lines more than origin/main — a .local/state
   chown fix; the only-if-missing logic is identical on both.)

   Verification checklist (all ✅)

   1. Trigger names / branch filters / buckets / included_files — ✅
      • opencode-staging-branch → push ^staging$ → gs://qbraid-codeq-staging/latest/linux-x64/codeq
      • opencode-prod-main → push ^main$ → gs://qbraid-codeq/latest/linux-x64/codeq
      • build-lab-base (^staging$) / build-lab-base-prod (^main$), both included_files = ["qbraid-lab-base/**"]
   2. gsutil cp …/codeq inlined in trigger, NOT chained to a codeq upload — ✅
      The download is an inline gsutil cp in the lab-base trigger's
      download-artifacts step. There is no google_storage_notification,
      Pub/Sub, Cloud Function, or Eventarc on the codeq bucket, and nothing
      programmatically runs build-lab-base[-prod] after an upload. The rebuild
      fires only on a qbraid-lab-base/** push (or manual trigger run). This is
      the Layer-1 gap, confirmed. (The build-lab-base hits in
      cloud-build-docker-stacks.tf/extensions.tf are step-ids inside the legacy
      V1 docker-stacks monolith — unrelated to the V2 trigger.)
   3. Helm singleuser.image tag: latest + pullPolicy: Always — ✅ in both
      values-staging.yaml and values-prod.yaml (singleuser image = lab-base).
   4. Runtime scripts + home-sync persistence — ✅
      • setup-codeq-config.sh is only-if-missing (if [ ! -f "$CONFIG_FILE" ]).
      • update-codeq-token.sh re-asserts (always) apiKey, baseURL,
        mcp.pod_mcp.command; sets mcp.pod_mcp.type/enabled only if absent;
        migrates dead model ids → current and defaults empty → qbraid/gemini-3.5-flash,
        preserving valid user models. It touches nothing else (not the permission
        block).
      • GCS home-sync confirmed: data-restore initContainer runs
        rclone sync gcs:qbraid-<env>-users/$JUPYTERHUB_USER/ /home/jovyan/ on every
        pod start, so ~/.config/codeq/config.json persists → only-if-missing means
        returning users never get template updates (Layer-3 gap).
   5. codeq --version the only on-pod way to identify the binary — ✅ (with
      nuance). Image is always :latest; no version file is baked in; manifest.json
      lives in the bucket, not the pod. codeq --version (wrapper → codeq-bin --version) is the only on-pod signal; OPENCODE_BUMP=1 makes builds report
      distinct versions, so it's a usable discriminator — but there is no
      at-a-glance external marker.

   Added precision (beyond the map's gist)

   • The codeq binary is also baked by build-lab-intel[-prod] and the disabled
     build-bma-lab-base trigger — any Layer-1 fix must cover every consumer.
   • Prod Cloud Build SA holds objectViewer on gs://qbraid-codeq-staging for
     cutover bridging (Phase B) — envs aren't strictly bucket-isolated during cutover.
   • Binary + wrapper are baked to /usr/local/bin (outside home) explicitly to
     dodge GCS home-sync — so the binary is only ever stale via Layers 1–2, never
     via home persistence; config (in home) is the only home-synced piece.

   Could not verify without live GCP access

   • Current contents/timestamps of gs://qbraid-codeq[-staging]/latest/ (is a newer
     binary sitting un-baked right now?).
   • Which digest lab-base:latest currently points to / last build time (Cloud Build history).
   • Whether triggers are currently enabled/connected (local.github_connection_ready).
   • The actual codeq version any running pod reports.
   • Whether a manual gcloud builds triggers run recently forced a rebuild.

   Deliverable

   The anchor doc is authored and committed locally as CODEQ_DEPLOYMENT.md on
   branch docs/codeq-deployment-anchor (off dev, commit acd919c). It is
   not pushed — say the word and I'll push it / open a PR into dev. Full
   content below.

   CODEQ_DEPLOYMENT.md (authored anchor)

   CodeQ deployment path (staging / prod Lab instances)

   How a change to qBraid/opencode reaches a JupyterLab singleuser pod as an
   updated codeq binary, and where staleness can creep in. This is the anchor
   document for the CodeQ-deployment wayfinder effort
   (map #16): every fact below is
   cross-checked against the live source of truth so downstream decision tickets
   build on correct facts.

   Scope: the GKE staging (qbraid-staging) and prod (qbraid-prod)
   Lab-instance path only. Mac/Windows/arm64 desktop & CLI distribution (the
   codeq-gcs-upload.yml GitHub Actions path) and BMA / multi-cloud AMI-bake
   instances are out of scope here.

   Sources of truth

   The GCP side is not owned by this repo. The authoritative files are:

   Concern	Repo · path
   Build codeq binary → upload to GCS	qbraid-infrastructure · terraform/environments/{staging,prod}/gcp/cloud-build-codeq.tf
   Download binary → bake into Lab image	qbraid-infrastructure · terraform/environments/{staging,prod}/gcp/cloud-build-qbraid-dstacks.tf
   Which image singleuser pods run	qbraid-infrastructure · kubernetes/clusters/gcp/gke_{staging,prod}_cluster/helm/jupyterhub/values-{staging,prod}.yaml
   Runtime config + token scripts, binary placement	qbraid-Dstacks · qbraid-lab-base/scripts/{setup-codeq-config,update-codeq-token,codeq-wrapper}.sh, qbraid-lab-base/Dockerfile

   Verified against: qbraid-infrastructure main @ 01b278b; qbraid-Dstacks
origin/staging (bbf58c3) and origin/main (69999a9) — the branches that
   deploy to staging and prod respectively.

   The 3-hop pipeline

   push to opencode           push to qbraid-Dstacks              new pod spawn
      staging/main    ─▶  (qbraid-lab-base/** changed)   ─▶     (pullPolicy: Always)
           │                        │                                  │
      HOP 1 · AUTO            HOP 2 · NOT AUTO                    HOP 3 · AUTO
      build binary,          download binary from GCS,           pull lab-base:latest,
      upload to GCS          bake into lab-base:latest           run baked binary
   

   Hop 1 — push → binary in GCS · automatic

   • Trigger opencode-staging-branch fires on push to ^staging$ of
     qBraid/opencode; opencode-prod-main fires on push to ^main$.
   • Each is a single Cloud Build: install deps → apply qBraid branding → build the
     single-file codeq binary (bun run build --single, OPENCODE_BUMP=1) →
     Trivy CLI-dependency scan → upload-gcs step (gsutil cp the binary) →
     upload-manifest.
   • Output:
     • staging → gs://qbraid-codeq-staging/latest/linux-x64/codeq (+ latest/manifest.json)
     • prod → gs://qbraid-codeq/latest/linux-x64/codeq (+ latest/manifest.json)
   • Only linux-x64 is produced here. manifest.json records
     {"version":"latest","timestamp":"…"} — it lives in the bucket, not on any pod.

   Hop 2 — binary in GCS → Lab image · NOT automatic (Layer 1 gap)

   • Trigger build-lab-base (staging, ^staging$) and
     build-lab-base-prod (prod, ^main$) on qBraid/qbraid-Dstacks, each
     narrowed by included_files = ["qbraid-lab-base/**"].
   • Their download-artifacts step inlines a plain
     gsutil cp gs://qbraid-codeq-staging/latest/linux-x64/codeq … (staging) /
     gs://qbraid-codeq/latest/linux-x64/codeq … (prod) into
     qbraid-lab-base/codeq_bin/, chmod +x, then the build step bakes it into
     …/jupyterhub/lab-base:latest (re-tagged lab-standard:latest).
   • This gsutil cp is an inline pull at build time — it is not chained to the
     codeq upload. There is no storage notification, Pub/Sub subscription, Cloud
     Function, or Eventarc watching the codeq bucket, and nothing programmatically
     runs the build-lab-base[-prod] trigger after an upload. The image rebuild
     fires only on a push to qbraid-Dstacks staging/main touching
     qbraid-lab-base/** (or a manual
     gcloud builds triggers run build-lab-base[-prod]).
   • Consequence (Layer 1): a freshly built codeq binary can sit in GCS
     indefinitely. It only reaches an image when some unrelated qbraid-lab-base/**
     change happens to trigger a rebuild, or someone manually re-runs the trigger.
     This is the core decoupling this effort exists to close.
   • The same binary is also baked by build-lab-intel[-prod]
     (foundation:py310 → lab-base:py310 → lab-intel; its included_files include
     qbraid-lab-base/**) and by the disabled build-bma-lab-base trigger — so
     the layer-1 fix must consider every consumer of the binary, not just lab-base.
   • Cutover nuance: the prod Cloud Build SA holds objectViewer on
     gs://qbraid-codeq-staging "so prod lab-base can consume the staging binary
     during cutover bridging (Phase B)." During cutover the environments are not
     strictly bucket-isolated.

   Hop 3 — Lab image → user pod · automatic for new pods only (Layer 2 gap)

   • singleuser.image in values-{staging,prod}.yaml:
     name: …/jupyterhub/lab-base, tag: latest, pullPolicy: Always.
   • Every newly-spawned pod pulls lab-base:latest and therefore runs whatever
     binary the current image baked in.
   • Consequence (Layer 2): already-running pods are not refreshed. A user
     keeps their old binary until they restart their server (new spawn → fresh
     pull). "Reached all users on an env" is not true until the last long-running
     pod is recycled.

   Runtime config & the binary on the pod

   Binary placement — outside home, never stale-from-sync

   The Dockerfile copies the binary to /usr/local/bin/codeq-bin and the wrapper
   to /usr/local/bin/codeq — explicitly not under ~/.local/bin, with the
   comment "which gets overwritten by GCS home sync." So the binary a user runs is
   always the image's baked binary; it is only ever stale via Layers 1–2, never via
   home persistence.

   codeq (the wrapper, codeq-wrapper.sh) runs update-codeq-token.sh silently,
   then execs /usr/local/bin/codeq-bin "$@".

   Home persistence — GCS sync (why config goes stale)

   The data-restore initContainer runs
   rclone sync gcs:qbraid-<env>-users/$JUPYTERHUB_USER/ /home/jovyan/ on every
   pod start. So ~/.config/codeq/config.json (and anything else in home) persists
   across pod restarts, restored from GCS.

   Layer 3 — config staleness

   On pod start, before-notebook.d/setup-qbraid.sh runs setup-codeq-config.sh
   then update-codeq-token.sh:

   • setup-codeq-config.sh — only-if-missing. It writes
     ~/.config/codeq/config.json only if the file does not already exist
     (if [ ! -f "$CONFIG_FILE" ]), else it logs "already exists" and leaves it
     untouched. Because the config is GCS-persisted (above), a returning user's
     config is never regenerated. New template fields added to this script
     (permissions, provider options, new MCP servers, model default) do not reach
     existing users. This is the Layer-3 gap. The template it would write pins
     model: qbraid/gemini-3.5-flash, baseURL: <ACCOUNT_URL>/api/ai/v1,
     apiKey seeded from QBRAID_ACCESS_TOKEN, a full permission allow-block, and
     the pod_mcp MCP server.

   • update-codeq-token.sh — the only self-healing path for existing configs.
     Runs on every pod start and every codeq launch (via the wrapper). On the
     existing config it re-asserts, via jq (sed fallback):

     • .provider.qbraid.options.apiKey ← QBRAID_ACCESS_TOKEN (fallback: ~/.qbraid/qbraidrc) — always
     • .provider.qbraid.options.baseURL ← <ACCOUNT_URL>/api/ai/v1 (fallback: qbraidrc url map; default https://account.qbraid.com) — always
     • .mcp.pod_mcp.command ← ["/usr/bin/python3","/opt/qbraid-mcp-server/server/main.py"] — always
     • .mcp.pod_mcp.type / .enabled ← "local" / true — only if absent (//=)
     • .model — migrates now-dead ids to current (e.g. gemini-3-flash→gemini-3.5-flash, gemini-3-pro→gemini-3.1-pro, claude-opus-4-6→claude-opus-4-8, claude-sonnet-4-5→claude-sonnet-4-6, grok-4.1-fast→gemini-3.5-flash); empty → qbraid/gemini-3.5-flash; a valid user-chosen model is preserved.

     It does not touch the permission block or any field outside those above.
     So the only config drift that self-heals for existing users is token, gateway
     URL, MCP command, and dead-model migration — nothing else.

   Version detection

   • The image is always tagged :latest; there is no per-build image tag, no pod
     label, and no version file baked into the image. manifest.json records a
     build timestamp but lives in the GCS bucket, not on the pod.
   • On a running pod, codeq --version is effectively the only way to tell which
     binary build a user is on. (codeq --version → wrapper → codeq-bin --version.) Because every build runs with OPENCODE_BUMP=1, distinct builds
     report distinct versions, so --version is a usable discriminator — but there
     is no at-a-glance external marker (image digest aside, which requires cluster
     access, not a user-visible signal).

   The three staleness layers (summary)

   Layer	Where	Mechanism	Auto-heals?
   1 binary → image	GCS → lab-base:latest	inline gsutil cp at image-build time; no chain from upload	❌ needs a qbraid-lab-base/** push or manual trigger run
   2 image → running pod	lab-base:latest → pod	pullPolicy: Always on spawn	⚠️ new pods only; running pods need a restart
   3 config persistence	GCS home-sync → config.json	setup-codeq-config.sh only-if-missing; update-codeq-token.sh re-asserts a fixed field set	⚠️ only token/baseURL/MCP-command/dead-model self-heal

   Verified but not confirmable without live GCP access

   The following depend on live-project state and could not be checked from source:

   • Current contents/timestamps of gs://qbraid-codeq[-staging]/latest/ — i.e.
     whether a newer binary is sitting un-baked right now.
   • Which image digest lab-base:latest currently points to and when it was last
     built (Cloud Build history).
   • Whether the triggers are currently enabled/connected in the live projects
     (local.github_connection_ready).
   • The actual codeq version any given running pod reports.
   • Whether any manual gcloud builds triggers run … has recently forced a rebuild.
5. Kenny-Heitritter

   closed this as completedon Jul 24, 2026