[deependujha]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #813
closes #943

thanks @kostasrigatos

Instead of patching cache env vars test by test, every test now gets its own cache dirs, so tests can't delete each other's files, even on parallel xdist workers.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82%. Comparing base (c6f075c) to head (6981042).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #952   +/-   ##
=====================================
  Coverage     82%     82%           
=====================================
  Files         72      72           
  Lines      16445   16445           
=====================================
+ Hits       13505   13507    +2     
+ Misses      2940    2938    -2     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

🟢 Approval recommended

Static review found no unresolved correctness issues in cache isolation or the updated tests; automated tests were not run.

No open issues 🎉

What changed in this PR

This PR addresses the parallel-test cache collisions described in #813 and broadens #943’s targeted approach to isolate cache directories across the test suite.

Changes:

• Adds per-test cache paths through environment variables inherited by spawned workers.
• Resolves LITDATA_CACHE_DIR at call time instead of import time.
• Removes shared-cache cleanup and worker grouping, and updates cache-path tests.

File	Description
tests/​utilities/​test_dataset_utilities.py	Tests environment overrides without module reloads and isolates cloud-default checks.
tests/​streaming/​test_parquet.py	Removes shared-cache cleanup usage and updates default-path assertions.
tests/​processing/​test_data_processor.py	Adds coverage for optimizer cache defaults and environment overrides.
tests/​conftest.py	Introduces automatic per-test cache isolation and removes shared-cache cleanup and grouping.
src/​litdata/​utilities/​dataset_utilities.py	Reads the cache-directory environment override at call time.
src/​litdata/​constants.py	Removes the import-time cache override constant.
pyproject.toml	Removes the obsolete cache-sharing marker registration.

_{Review effort: Balanced}

[bhimrazy]

LGTM,
Added some minor nits from cc review.