Skip to content

Intermittent CI failures in cache::list_files_cache::tests::test_cache_with_ttl_and_lru ( seems flaky) #19114

Description

@alamb

Describe the bug

@adriangb reports on discord:

cache::list_files_cache::tests::test_cache_with_ttl_and_lru is intermittently failing on CI

To Reproduce

Seems like it's failing on main: https://github.com/apache/datafusion/actions/runs/19962057560/job/57244600157#step:4:2618

failures:

---- cache::list_files_cache::tests::test_cache_with_ttl_and_lru stdout ----

thread 'cache::list_files_cache::tests::test_cache_with_ttl_and_lru' (146096) panicked at datafusion/execution/src/cache/list_files_cache.rs:618:9:
assertion failed: cache.contains_key(&path3)
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/f8297e351a40c1439a467bbbb6879088047f50b3/library/std/src/panicking.rs:698:5
   1: core::panicking::panic_fmt
             at /rustc/f8297e351a40c1439a467bbbb6879088047f50b3/library/core/src/panicking.rs:75:14
   2: core::panicking::panic
             at /rustc/f8297e351a40c1439a467bbbb6879088047f50b3/library/core/src/panicking.rs:145:5
   3: test_cache_with_ttl_and_lru
             at ./src/cache/list_files_cache.rs:618:9
   4: {closure#0}
             at ./src/cache/list_files_cache.rs:596:37
   5: call_once<datafusion_execution::cache::list_files_cache::tests::test_cache_with_ttl_and_lru::{closure_env#0}, ()>
             at /rustc/f8297e351a40c1439a467bbbb6879088047f50b3/library/core/src/ops/function.rs:250:5
   6: core::ops::function::FnOnce::call_once
             at /rustc/f8297e351a40c1439a467bbbb6879088047f50b3/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Also failed on a PR with unrelated changes
https://github.com/apache/datafusion/actions/runs/19966253699/job/57258712946?pr=19094#step:4:2618

Expected behavior

test should always pass (or fail) not intermittently pass

Additional context

The test is here
https://github.com/apache/datafusion/blame/a5fc3c704b261f3a3ccf35089b93d8424901ba96/datafusion/execution/src/cache/list_files_cache.rs#L596

It was added in d150af7 / #18855 from @BlakeOrth

I suspect the runner is so slow that the ttl expires prematurely

Activity

  1. added theissue type on Dec 5, 2025
  2. added
    bugSomething isn't working
    on Dec 5, 2025
  3. alamb commented on Dec 5, 2025

    @alamb
    ContributorAuthor

    Suggested course of action:

    1. Disable the test temporarily to unblock CI on main
    2. Fix the test (probably by mocking out where the current time comes from, rather than using the real system clock)
  4. alamb commented on Dec 5, 2025

    @alamb
    ContributorAuthor

    Suggestion for implementation: #18855 (comment)

    One classic way is to pass a template / object to the cache to provide the current time, rather than calling std::instant::now() directly

    Then the tests can use a mock version of the time provider (that is explicitly controlled) and the normal code uses std::Instant::now

    The downside of this approach is that it does mean what is tested has some differences than what is in production

  5. BlakeOrth commented on Dec 5, 2025

    @BlakeOrth
    Contributor

    I will plan on allocating time to address this early next week unless someone beats me to it over the weekend.

  6. added a commit that references this issue on Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions