av1: decode frames smaller than the sequence maximum at their real size

[yevhen-sychov]

av1: decode frames smaller than the sequence maximum at their real size

What you see

With some AV1 streams, hardware decode through this driver starts out fine and
then falls apart: a few seconds after each keyframe, moving parts of the picture
turn into broken blocks, and the damage keeps spreading the more things move
until the next keyframe arrives. Static content can look almost normal, which
makes it easy to mistake for a network or encoder problem.

We hit it in Chromium WebRTC loopback and screen sharing with a hardware AV1
encoder, but it is not Chromium-specific: ffmpeg -hwaccel vaapi shows exactly
the same corruption on the same files, while libdav1d and NVIDIA's own Vulkan
Video decoder (same GPU) decode them perfectly.

Which streams are affected

AV1 lets a stream declare a maximum frame size in its sequence header and then
code individual frames smaller than that (frame_size_override_flag). Hardware
encoders do this all the time. NVENC, for example, works in 16-pixel-aligned
blocks, so for 1080p content it declares a 1920x1088 maximum and codes every
frame at 1920x1080 (and 3840x2176 vs 3840x2160 for 4K).

Streams whose frames are coded at the full maximum size (for example most
software-encoded content) are not affected, which is probably why this has
gone unnoticed.

Why it happens

Players correctly create the VA context and surfaces at the sequence maximum.
The driver then assumed every frame was that size as well:

1. copyAV1PicParam told NVDEC the current frame was the context size
   (1920x1088), and gave every reference frame the size of its surface
   (also 1920x1088), instead of the sizes the frames were really coded at
   (1920x1080). The first frames after a keyframe still come out right, but
   prediction runs against the wrong frame geometry, so small errors appear
   wherever there is motion and compound frame after frame.

2. Once the right sizes are passed, a second issue shows up: NVDEC does not
   crop a smaller frame out of the decoder output, it stretches it to fill
   the display area the decoder was created with. A 1080-line frame would come
   out scaled to 1088 lines.

The first point was found by recording the exact CUVIDAV1PICPARAMS this
driver and ffmpeg's own NVDEC AV1 hwaccel send for the same stream: the only
fields that differed were the frame and reference sizes.

What the patch changes

• Real sizes in the picture parameters. The current frame gets its coded
  size; that size is remembered on the surface the frame is decoded into, and
  each reference frame is described with its remembered size instead of its
  surface size. The picture parameters now match ffmpeg's NVDEC hwaccel.

• Display area follows the frame size. When a frame's size differs from the
  decoder's display area, the display area is changed with
  cuvidReconfigureDecoder. The frame is placed unscaled in the top-left of the
  unchanged, surface-sized target, so the existing copy into the surface keeps
  working as before.

  Two details worth knowing when reviewing:

  • NVDEC refuses a reconfigure before the decoder has decoded anything, so the
    first change is applied right after the first picture is decoded and before
    that picture is handed to the resolve thread (the display area takes effect
    when a frame is mapped, not when it is decoded).
  • If the size changes again later in a stream, frames already queued for
    output are allowed to finish before the display area changes, so they are
    not cropped with the new size.

Streams whose frames are all coded at the maximum size never request a display
area change and take exactly the same path as before.

How it was tested

RTX GPU, NVIDIA driver 610.57.04, direct backend. Each stream was decoded
through this driver with ffmpeg -hwaccel vaapi and compared frame by frame
against libdav1d:

Stream	Before	After
1920x1080 frames, 1920x1088 sequence maximum	59 / 150 frames bit-exact	150 / 150
3840x2160 frames, 3840x2176 sequence maximum	30 / 90	90 / 90
AV1 frames equal to the sequence maximum (av1_nvenc)	150 / 150	150 / 150
H.264 decode	—	byte-identical to before

No reconfigure failures or CUDA errors were logged. Also confirmed by eye in a
Chromium WebRTC AV1 loopback with heavy motion: before, the picture degraded
into blocks within seconds; after, it stays clean.

Not covered: a stream whose frame size changes in the middle of playback (no
such sample was available). The code path for it is described above.

Reproducing it yourself

You need a stream whose frames are smaller than the sequence maximum. The
simplest way to check an existing file:

ffmpeg -i input.ivf -c:v copy -bsf:v trace_headers -f null - 2>&1 \
  | grep -E "max_frame_height_minus_1|frame_size_override_flag|frame_height_minus_1" | head

frame_size_override_flag = 1 with frame_height_minus_1 smaller than
max_frame_height_minus_1 means the stream is affected.

Then compare the hardware decode with libdav1d (1080p example):

ffmpeg -c:v libdav1d -i input.ivf -f rawvideo -pix_fmt yuv420p dav1d.yuv
LIBVA_DRIVER_NAME=nvidia ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi -i input.ivf \
  -vf 'hwdownload,format=nv12' -f rawvideo -pix_fmt yuv420p vaapi.yuv
ffmpeg -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i vaapi.yuv \
       -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i dav1d.yuv \
       -lavfi psnr=stats_file=psnr.txt -f null -
grep -c 'psnr_y:inf' psnr.txt   # frames that match exactly

Without the patch the count stops matching roughly 30 frames after every
keyframe; with it, every frame matches.

🤖 Generated with Claude Code