From 89e534942f0d2023854e3c1e61403783b357b1d4 Mon Sep 17 00:00:00 2001
From: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
Date: Wed, 14 Oct 2026 05:30:30 +0000
Subject: [PATCH] [None][infra] Fix missing perf folders in GB300 PerfSanity
 test results (#20261)

PerfSanity stages on oci-jhb and oci-aga (the GB300 stages) do not
upload their perf folders, the per-test aggr_upload-*/disagg_upload-*
folders with the server logs and cache_transceiver_precheck data.
Nothing is logged.

Since #14138 the sbatch path lists these folders with

    find '<ws>' ... -name 'aggr*' -o -name 'disagg*' ... -print 2>/dev/null

Both consumers paste that command between single quotes, which strips
its inner single quotes, and sshd runs it under svc_tensorrt's login
shell. On oci-jhb and oci-aga that shell is csh. (oci-hsg also uses
csh, but its sshd ForceCommand runs the command with bash.) Under csh
the bare globs fail with "No match", and 2>/dev/null would become an
extra argument plus a stdout redirect. The error is discarded locally,
so the list comes back empty.

Run only "find <ws> -mindepth 1 -maxdepth 1 -type d" remotely and keep
the aggr/disagg folders with a local grep. Move the NFS cache-refresh
command's redirects to the local side as well; under csh they wrote a
stray file named 1 into svc_tensorrt's home. Also drop 2>/dev/null from
the slurm_job_id.txt fallback read, which under csh always returned an
empty job ID.

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
(cherry picked from commit 5828d4f9f1df22e4d73f0eae5c4a62a87e2a12fa)
---
 jenkins/L0_Test.groovy | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/jenkins/L0_Test.groovy b/jenkins/L0_Test.groovy
index d5b150f3dd4a..72982e926461 100644
--- a/jenkins/L0_Test.groovy
+++ b/jenkins/L0_Test.groovy
@@ -2450,9 +2450,11 @@ def runLLMTestlistWithSbatch(pipeline, platform, testList, config=VANILLA_CONFIG
                     // `|| true` so a missing file (submission never wrote it) yields
                     // an empty string and reaches the typed throw below, instead of a
                     // generic "script returned exit code 1" the classifier can't act on.
+                    // No `2>/dev/null`: under a csh login shell it would send stdout to
+                    // /dev/null and always return an empty job ID.
                     slurmJobId = Utils.exec(
                         pipeline,
-                        script: Utils.sshUserCmd(remote, "\"cat ${jobWorkspace}/slurm_job_id.txt 2>/dev/null || true\""),
+                        script: Utils.sshUserCmd(remote, "\"cat ${jobWorkspace}/slurm_job_id.txt || true\""),
                         returnStdout: true,
                         numRetries: 3
                     ).trim()
@@ -2616,8 +2618,13 @@ def runLLMTestlistWithSbatch(pipeline, platform, testList, config=VANILLA_CONFIG
                 def sshStatCmd = Utils.sshUserCmd(remote, "\"srun --overlap ${srunAccountArg}--quiet --jobid='${slurmJobId}' --ntasks=1 stat -c %Y '${remoteWorkspaceTrk}/results.xml' || echo 0\"")
                 def scpXmlCmd = scpFromRemoteCmd(remote, "${remoteWorkspaceTrk}/results*.xml", "${stageName}/")
                 def scpUnfinishedCmd = scpFromRemoteCmd(remote, "${remoteWorkspaceTrk}/unfinished_test.txt", "${stageName}/")
-                def sshRefreshCacheCmd = Utils.sshUserCmd(remote, "\"ls '${remoteWorkspaceTrk}/' > /dev/null 2>&1; ls -la '${remoteWorkspaceTrk}/results.xml' > /dev/null 2>&1 || true\"")
-                def sshListPerfCmd = Utils.sshUserCmd(remote, "\"find '${remoteWorkspaceTrk}' -maxdepth 1 -type d \\( -name 'aggr*' -o -name 'disagg*' \\) -print 2>/dev/null || true\"")
+                // The sh blocks below paste these two commands between single quotes, so they
+                // must not contain single quotes (a `bash -c '...'` wrapper would break too).
+                // sshd runs the remote part under svc_tensorrt's login shell, which is csh on
+                // some clusters (e.g. oci-jhb, oci-aga); csh rejects unmatched globs and has no
+                // `2>`. So keep redirects and filters on the local side of the ssh call.
+                def sshRefreshCacheCmd = Utils.sshUserCmd(remote, "\"ls -la ${remoteWorkspaceTrk}/ ${remoteWorkspaceTrk}/results.xml\"") + " > /dev/null 2>&1 || true"
+                def sshListPerfCmd = Utils.sshUserCmd(remote, "\"find ${remoteWorkspaceTrk} -mindepth 1 -maxdepth 1 -type d\"") + " | grep -e /aggr -e /disagg"
                 def scpPerfTemplate = scpFromRemoteCmd(remote, "PERF_FOLDER_PLACEHOLDER", "${stageName}/")
                 sh "rm -f ${pytestDoneFile}"
                 withCredentials([usernamePassword(