Skip to content

Resolve an indirect offset inside a named list from the use site - #3602

Merged
ESultanik merged 1 commit into
masterfrom
fix/indirect-offset-inside-named-test
Sep 18, 2026
Merged

ESultanik merged 1 commit into
masterfrom
fix/indirect-offset-inside-named-test

Conversation

@ESultanik

Copy link
Copy Markdown
Collaborator

A test nested under a name block counts its offsets from wherever the use that dispatched it matched. NamedAbsoluteOffset does that, but MagicTest.__init__ applied it only to a test whose own offset is an AbsoluteOffset. An indirect offset is not one, so the position it reads its pointer from was left counting from the start of the file, and every (N.x) inside a named list read the wrong bytes.

What it looked like

magic_defs/cafebabe:29 declares >(8.L) indirect x, which should read the architecture's file offset 8 bytes into the fat_arch record the use landed on. For /usr/bin/nohup:

pointer at absolute 16 = 16384      the first architecture
pointer at absolute 36 = 49152      the second
what was read          = 16777223   0x01000007, CPU_TYPE_X86_64

Both architectures dispatched to the same out-of-bounds offset, so neither reported anything:

file:    Mach-O universal binary with 2 architectures: [x86_64:\012- Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>] [...]
before:  Mach-O universal binary with 2 architectures: [ x86_64:] [ arm64e (caps: 0x2):]
after:   Mach-O universal binary with 2 architectures: [ x86_64: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>] [...]

The fix

rebase_in_named_test walks the offset instead of type-testing it. It rebases a position and leaves a distance alone: what a relative (&) offset wraps is a distance from the previous match, so rebasing it would add the use site twice.

The regression test is self-contained rather than reliant on a Mach-O binary. It puts the pointer a named list declares at offset 12 and a decoy at offset 4, which is what an offset counted from the start of the file reads instead. file 5.48 reports base [:nested for the same input.

What this does not fix

Two differences from file remain on a universal binary, both in how the message is assembled rather than where it reads:

  1. PolyFile writes [ x86_64, libmagic writes [x86_64. A \b following a use is not suppressing the separating space.
  2. libmagic emits its match separator before each nested verdict, so it writes :\012- Mach-O 64-bit ... where PolyFile writes : Mach-O 64-bit ....

So this alone moves the numbers by one file, not by forty. Over the same 2,000 real files, strongest-match agreement goes from 1765/1998 to 1766/1998 and containment from 1922/1998 to 1923/1998. The Mach-O descriptions now carry the right content but still do not match byte for byte, so the differential does not count them. Closing that cluster needs the two items above, which I will file separately.

Full suite: 384 passed, 1,511 subtests. Blocking flake8: 0. The test fails with the change reverted.

🤖 Generated with Claude Code

A test nested under a `name` block counts its offsets from wherever the
`use` that dispatched it matched. `NamedAbsoluteOffset` does that, but
only for a test whose own offset is an `AbsoluteOffset`. An indirect
offset is not one, so the position it reads its pointer from was left
counting from the start of the file.

Every `(N.x)` inside a named list therefore read the wrong bytes. On a
Mach-O universal binary, `magic_defs/cafebabe:29` declares `>(8.L)
indirect x`, which should read the architecture's file offset 8 bytes
into the fat_arch record the `use` landed on. It read absolute offset 8
instead, which is the CPU type:

    /usr/bin/nohup
      pointer at absolute 16 = 16384       the first architecture
      pointer at absolute 36 = 49152       the second
      what was read          = 16777223    0x01000007, CPU_TYPE_X86_64

Both architectures dispatched to the same out-of-bounds offset, so
neither reported anything and the brackets came out empty:

    file:     ... [x86_64:\012- Mach-O 64-bit x86_64 executable, flags:<...>] ...
    before:   ... [ x86_64:] [ arm64e (caps: 0x2):]
    after:    ... [ x86_64: Mach-O 64-bit x86_64 executable, flags:<...>] ...

`rebase_in_named_test` walks the offset instead of type-testing it. It
rebases a position and leaves a distance alone: what a relative (`&`)
offset wraps is a distance from the previous match, and rebasing it would
add the `use` site twice.

Two differences from `file` remain on this input, both in how the message
is assembled rather than where it reads: PolyFile writes `[ x86_64` where
libmagic writes `[x86_64`, and libmagic emits its match separator before
each nested verdict. Over 2,000 real files this fix alone moves the
strongest-match agreement by one file, because the Mach-O descriptions
now hold the right content but still do not match byte for byte.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RG45aqyyCLkL5b1UB5GWd7
@ESultanik ESultanik self-assigned this Sep 18, 2026
@ESultanik
ESultanik merged commit 6aaeee1 into master Sep 18, 2026
14 checks passed
@ESultanik
ESultanik deleted the fix/indirect-offset-inside-named-test branch September 18, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant