Skip to content

hyperloom custom workload: a DFlash arm names no drafter, so the 50.87 best of record cannot say which of this box's two drafter quants produced it #977

Closed

Description

@bong-water-water-bong

Class: the artifact does not describe the measurement (same as #892, #930, #932, #947, #954 — all fixed).

HL_DFLASH is not a configuration on its own: serve_lever_args() hands --dflash the file HL_DFLASH_GGUF names,
defaulting to the relative path dflash.gguf (so the run's working directory decides it too), and HL_MTP_HEAD
substitutes an external MTP head for the model's own. levers() — the only place a result echoes its configuration —
lists HL_DFLASH and HL_MTP_MAX but neither HL_DFLASH_GGUF nor HL_MTP_HEAD.

This model has two drafter quants on this box (~/models/qwen38/DFlash2/{q8_0,q4k}.gguf, 2,056,414,560 B and
1,207,231,840 B), so the gap is not academic — and the tuner itself treats the pair as one configuration: it drops a
drafter-quant variant because overwriting HL_DFLASH_GGUF "would make its number incomparable to the baseline"
(hl-workspace-26030/... orchestrator log, 2026-10-30 18:14:39).

The arm of record, 50.87 (HL_DFLASH=1 HL_MTP_MAX=6), is exactly this shape. Its artifact:

$ python3 -c "import json;d=json.load(open('.../benchmark_custom_20261101_003158/inferencex_result.json'));print(sorted(d['bench_summary']),d['bench_summary']['levers'])"
['levers','placement','pp512_tok_s','runner_type','score_geomean','serve_tok_s','tg128_tok_s'] {'HL_MTP_MAX': '6', 'HL_DFLASH': '1'}

So which drafter produced 50.87 is recoverable only from serve.log line 29 of that session
(loading draft model '/home/bcloud/models/qwen38/DFlash2/Qwen3.8-27B-DFlash2-q8_0.gguf') — or from the session's
config.yaml, which is Hyperloom's file, not the measurement's. Until #892 a second arm in the same RESULT_DIR
overwrote that serve.log; 50 HL_DFLASH-carrying artifacts (25 distinct runs x benchmark_report.json + inferencex_result.json) exist in that workspace and none of them names its drafter.

Same class, smaller: the pp512/tg128 medians are over -r HL_REPS repetitions (default 3) and the result never says how
many, so two numbers a couple of percent apart cannot be told apart from a rep-count change.

Wanted: every result names the drafting files it really used (resolved path + hash, and whether the path came from the
environment or the default) and the repetition count behind its medians.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions