[BUG] S3: object returns 404 right after a successful Put (dir cache overwritten by a lagging List, drivers without Getter e.g. BaiduNetdisk) / S3 上传成功后立即 Head 返回 404（目录缓存被滞后的 List 覆盖） #3207

Please confirm the following

• I have read and agree to AGPL-3.0 Section 15 .
  The program is provided "as is" without any warranties; you bear all risks of using it.

• I have read and agree to AGPL-3.0 Section 16 .
  The copyright holders and distributors are not liable for any damages resulting from the use or inability to use the program.

• I confirm my description is clear, polite, helps developers quickly locate the issue, and complies with community rules.

• I have read the OpenList documentation.

• I confirm there are no duplicate issues or discussions.

• I confirm this is an OpenList issue, not caused by other reasons (such as network, dependencies, or operation).

• I believe this issue must be handled by OpenList and not by a third party.

• I confirm this issue is not fixed in the latest version.

• I have not read these checkboxes and therefore I just ticked them all, Please close this issue.

OpenList Version (required)

v4.2.7 (3b28270a, current main at time of writing / 撰写时的最新 main)

Storage Driver Used (required)

BaiduNetdisk (百度网盘), accessed through the built-in S3 server. The bug is in internal/op, so it should affect any driver that does not implement driver.Getter and whose List can lag behind a write.

Bug Description (required)

Symptom: PutObject returns 200, but a HeadObject / GetObject on the same key right afterwards sometimes returns 404 (NoSuchKey). Once that happens, the key keeps returning 404 until the directory cache expires (cache_expiration) or someone refreshes the directory by hand.

This breaks backup tools that write an object and then read it back straight away. In our case on 2026-10-18 it broke Velero/Kopia uploads, and a mysqldump job failed when it checked for the complete.json it had just uploaded.

Steps to reproduce

1. Mount BaiduNetdisk with caching enabled (cache_expiration > 0) and enable the S3 server.
2. Send many PutObject requests to one prefix, each followed immediately by HeadObject on the same key. Running a few ListObjects on that prefix at the same time makes it happen more often.
3. Some HeadObject calls return 404 even though their PutObject succeeded, and the object is really in the netdisk.

Expected: an object that OpenList itself just wrote can be read back right away (read-after-write), like on AWS S3.
Actual: intermittent 404. It lasts until the cache expires.

Root cause (links point to v4.2.7)

• op.Put (fs.go#L683-L697) adds the new object to the dir cache only if that directory is already cached, using cache.UpdateObject. If it is not cached, nothing is recorded. PutURL behaves the same way (L753-L764).
• op.list (fs.go#L108) replaces the whole directory entry with Cache.dirCache.SetWithTTL(key, newDirectoryCache(files), ...). Suppose that List began before the upload finished, or the backend's list is eventually consistent (Baidu often omits a just-created file for a short time). Then it drops the entry Put just added.
• op.Get (fs.go#L165-L213): BaiduNetdisk has no Getter, so Get relies on the parent directory's listing. If the dir cache exists but the name is missing, Get returns ObjectNotFound without listing again. If the dir cache is missing, Get lists the directory, and a lagging List then caches a page without the new file. Either way the 404 sticks until the TTL expires. S3 HeadObject reaches this through fs.Get → op.Get (server/s3/backend.go#L120).

Related: #1725 (the same "new file missing after upload" symptom seen from the web UI; the maintainer comments there note the netdisk side lags behind writes). #3086 fixes a similar read gap during replace, but only for Google Drive; this issue covers the generic dir-cache path that every driver goes through.

There are two races:

• A: the dir is cached, a List (refresh or concurrent) takes its snapshot before Put finishes and stores it after Put's UpdateObject, so the new entry is lost.
• B: the dir cache expired or was invalidated during the upload, so Put records nothing, and the next List returns a lagging page without the file.

Proposed fix: keep a small in-memory "pending" record next to the dir cache, the way rclone VFS adds just-written files to its directory and merges them into later listings:

• After a successful Put/PutURL (and Move/Rename on the target side), record a pending-add for (dirKey, name) with a short TTL (5 min). Remove, Move and Rename on the source side record a pending-delete.
• In op.list, before SetWithTTL, merge the remote result with the pending records. Pending-adds the remote list does not show yet stay in the cache. Once the remote list shows the name, the pending-add is dropped (pending-until-seen). Names with a pending-delete are filtered out until the remote list stops returning them.
• op.Get checks pending-adds when the cached listing does not contain the name.
• It does not add refresh-on-miss: a miss never triggers an extra List, so a genuinely missing key still gets a 404 without extra API calls, and drivers that implement Getter are unaffected. The record is only in memory, and ClearAll also clears it.

I have this working on top of v4.2.7 with unit tests for both races and plan to open a PR soon.

Logs (required)

Unit-test reproduction against unmodified v4.2.7. It uses a fake driver with no Getter whose List can lag behind Put:

--- FAIL: TestReproA_RefreshListOverwritesPut (0.20s)
    repro_put_get_test.go:105: BUG REPRODUCED: Put OK but immediate Get -> ObjectNotFound
--- FAIL: TestReproB_CacheMissDuringPut (0.00s)
    repro_put_get_test.go:138: BUG REPRODUCED: Put OK but immediate Get -> ObjectNotFound

With the proposed change these tests pass, along with the lagging-List, true-404 and pending-delete cases.

Production stress test with the change (BaiduNetdisk via the S3 server, Put followed immediately by Head): 300/300 HeadObject returned 200 and none returned 404. That was 120 Puts with 12 workers, 80 Puts with 12 workers while the same prefix was being listed (126 List calls), and 100 Puts with 16 workers. Before the change, our production backup jobs hit intermittent 404s after successful Puts. See the A/B results below for before/after numbers.

中文： 上面是在未修改的 v4.2.7 上用单元测试复现的结果（模拟一个没有 Getter、List 会滞后于 Put 的驱动），两种竞态都复现了 404；应用修复后这些测试以及滞后 List、真实 404、pending-delete 等用例全部通过。生产环境压测（百度网盘 + S3，Put 后立即 Head）：300 次 HeadObject 全部 200、0 次 404，包括 120 次（12 并发）、80 次（12 并发，同时对该前缀 List 126 次）和 100 次（16 并发）。修复前生产备份任务会间歇性出现上传成功后 404。修复前后的对比数据见下方 A/B 测试。

Production evidence and an A/B run on BaiduNetdisk (added 2026-10-19)

Real-world occurrences (unpatched official image, cache_expiration = 30 min). Taken from OpenList's own access log:

• Velero/Kopia blob, 2026-10-16. The PUT returned 200 after 13.3 s. Over the next 3 minutes, 5 different backup pods sent 7 GETs for that key, and all 7 returned 404 in 0.3–1.3 ms, so they were answered from the directory cache without calling the driver. From then on GET returned 200, and the object still exists with LastModified equal to the PUT time. In that window, 7 Velero PodVolumeBackups failed with BLOB not found for exactly this blob ID, and the nightly backup ended PartiallyFailed.
• Backup script, 2026-10-18. complete.json PUT returned 200 after 13.3 s. A HEAD 3 s later returned 404 in 198 µs, again from cache, and the job failed. The same key returned HEAD/GET 200 later that day.

A/B run. I set cache_expiration to 1 min so that cache rebuilds happen during a short test. The load was 14 workers doing Put then Head, plus concurrent List and mkdir in the same directories. I re-checked every 404 later.

Head latency was unchanged at about 96/100 ms (p50/p95).

Out of scope. Separately, 63 PUTs in the fix runs (and 1 in the unpatched run) returned 200, but the file never landed. These came in bursts next to Baidu errno: 31500. A fresh refresh=true List through OpenList and through Baidu's own file-list API, done after the cache and pending windows had passed, did not contain them. In these cases Baidu's create call reports success without a path, and the driver names the object .. This is a backend/driver issue that this fix does not address. A minor side effect: with the fix, that phantom . entry can stay in the listing for up to 5 minutes; without the fix it stays until the cache expires.

生产环境证据与百度网盘 A/B 测试（2026-10-19 补充）

真实发生的案例（官方未修复镜像，cache_expiration = 30 分钟）。 来自 OpenList 自身的访问日志：

• Velero/Kopia blob，2026-10-16。 PUT 耗时 13.3 s 返回 200。之后 3 分钟内，5 个不同的备份 Pod 对该 key 发了 7 次 GET，全部返回 404，耗时 0.3–1.3 ms，即直接由目录缓存应答，没有调用驱动。此后 GET 都返回 200，对象至今存在，LastModified 等于 PUT 时间。同一时间窗口内有 7 个 Velero PodVolumeBackup 因该 blob ID BLOB not found 失败，当晚备份结果为 PartiallyFailed。
• 备份脚本，2026-10-18。 complete.json PUT 耗时 13.3 s 返回 200。3 秒后 HEAD 返回 404，耗时 198 µs，同样来自缓存，导致任务失败。当天晚些时候同一 key 的 HEAD/GET 返回 200。

A/B 测试。 把 cache_expiration 设为 1 分钟，让短时间测试内也会发生缓存重建。负载为 14 个并发 worker 执行 Put 后 Head，同时在同一目录下并发 List 和 mkdir；所有 404 事后都做了复查。

Head 延迟不变，p50/p95 约 96/100 ms。

不在本修复范围内。 另外，修复版两轮中有 63 次 PUT（未修复版 1 次）返回 200，但文件实际没有写入。这些都成批出现，同时伴随百度 errno: 31500。在缓存和 pending 窗口过后，通过 OpenList（refresh=true）和百度自身的文件列表 API 重新 List，都没有这些文件。这种情况下百度的 create 接口返回成功但没有 path，驱动把对象命名为 .。这是后端/驱动的问题，本修复不处理。一个小副作用：使用修复版时，这个虚假的 . 条目最多会在列表中保留 5 分钟；未修复时则保留到缓存过期为止。

Configuration File Content (required)

"s3": { "enable": true, "port": 5246, "ssl": false }

Storage: BaiduNetdisk, cache enabled (cache_expiration > 0), S3 bucket mapped to a folder on it. Secrets omitted.

存储：百度网盘，已开启缓存（cache_expiration > 0），S3 bucket 映射到其中一个目录。密钥已省略。

Reproduction Link (optional)

A PR with the fix and the tests above will follow. / 包含修复和上述测试的 PR 随后提交。

AI Generated Content

• I used AI tools to generate this content
• I did not use AI tools to generate this content

AI model used

Grok Bot