docs: document the kernels cache

docs/source/_toctree.yml

@@ -15,6 +15,8 @@
       title: Lock kernel versions
     - local: env
       title: Environment variables
+    - local: cache
+      title: Kernels cache
     - local: integrating-kernels
       title: Projects using kernels
     - local: migration

docs/source/cache.md

@@ -0,0 +1,242 @@
+# Kernels cache
+
+`kernels` downloads a kernel build from the Hub the first time you load it and
+then loads it from a local cache. Later loads of the same kernel, revision and
+build variant are served from that cache, so the kernel files are downloaded
+only once.
+
+The kernels cache is a regular Hugging Face Hub cache: kernel repositories live
+in the same directory as `models--…` and `datasets--…` repositories, and they
+use the same layout (`refs/`, `blobs/`, `snapshots/`). This page describes how
+the cache is configured, what is stored in it, and how to use it when the Hub
+is not reachable.
+
+## Cache location
+
+The cache directory is resolved in this order, first match wins:
+
+| Setting                     | Cache directory                          |
+| --------------------------- | ---------------------------------------- |
+| `KERNELS_CACHE`             | `$KERNELS_CACHE`                         |
+| `HF_HUB_CACHE`              | `$HF_HUB_CACHE`                          |
+| `HUGGINGFACE_HUB_CACHE`     | `$HUGGINGFACE_HUB_CACHE`                 |
+| `HF_HOME`                   | `$HF_HOME/hub`                           |
+| None of the above           | `$XDG_CACHE_HOME/huggingface/hub`, or `~/.cache/huggingface/hub` when `XDG_CACHE_HOME` is not set |
+
+`KERNELS_CACHE` is the only setting that is specific to `kernels`. Set it when
+you want kernel builds somewhere other than the cache of your models, for
+example on a fast local disk of a GPU machine, on a shared volume, or in a
+directory that is baked into a container image:
+
+```bash
+export KERNELS_CACHE=/mnt/fast-disk/kernels
+```
+
+When `KERNELS_CACHE` is set, it is used only by `kernels`. Other Hugging Face
+tools keep using the Hub cache, so kernels stored in `KERNELS_CACHE` are not
+visible to them, and `hf cache` commands do not list them.
+
+A few details on how the setting is read:
+
+- A leading `~` is expanded to your home directory.
+- An empty value counts as unset.
+- The variable is read on every call, not at import time. You can change
+  `KERNELS_CACHE` (or `os.environ["KERNELS_CACHE"]`) between two `get_kernel`
+  calls and the second call uses the new directory.
+
+Kernels are downloaded with the same credentials as any other Hub download:
+`HF_TOKEN`, or the token written by `hf auth login`. A private kernel
+repository is cached like any public kernel, so on a shared machine keep the
+cache directory readable only by you.
+
+If you use a Hub mirror through `HF_ENDPOINT`, kernels are downloaded from the
+mirror; the cache location is unaffected.
+
+## On-disk layout
+
+Kernel repositories use the `kernels--<org>--<name>` prefix. A cache holding
+one version of [`kernels-community/activation`](https://huggingface.co/kernels-community/activation)
+looks like this:
+
+```text
+<cache-dir>/
+├── kernels--kernels-community--activation/
+│   ├── refs/
+│   │   ├── main
+│   │   └── v1
+│   ├── blobs/
+│   │   └── 9f2c1d…            # file contents, addressed by hash
+│   └── snapshots/
+│       └── 4d1a8e…/           # one directory per downloaded commit
+│           └── build/
+│               └── torch28-cxx11-cu128-x86_64-linux/
+│                   ├── metadata.json
+│                   └── activation/
+│                       └── __init__.py
+└── .verified-kernels/
+    ├── digest/v1/
+    └── signature/v1/
+```
+
+### `snapshots/`
+
+Each downloaded revision gets one snapshot directory, named after its commit.
+The directory is a checkout of that commit that only contains the files that
+were actually downloaded. Kernels are downloaded by commit, and `kernels` only
+fetches the build variant that matches your system (`build/<variant>/*`), so a
+snapshot typically holds a single variant even when the repository has many.
+
+To populate more than one variant, use [`kernels download`](cli-download.md)
+with `--all-variants`, which is useful for building a cache that is shared
+between machines with different hardware or PyTorch versions.
+
+### `blobs/`
+
+File contents, addressed by their hash. Entries in a snapshot are symlinks into
+`blobs/` (copies on systems where symlinks are not available), so two revisions
+that share a file also share its blob and the space it takes.
+
+### `refs/`
+
+Maps a branch or tag, such as `v1`, to the commit it points at. Since kernels
+are downloaded by commit, `snapshot_download` would not create these refs on its
+own. `kernels` therefore records the ref itself when a kernel is fetched by
+version or revision, which is what makes `get_kernel(..., version=1)` resolvable
+later without a Hub request (see [offline use](#offline-use)).
+
+Refs are a local mirror of the Hub branches and tags and can be deleted: they
+are rewritten the next time the kernel is downloaded while online. Deleting a
+ref only breaks resolving that branch or tag from the local cache.
+
+### `.verified-kernels/`
+
+Verification receipts. A kernel that has been verified once gets a receipt, so
+that its files are not hashed and its signature is not verified again on every
+load. Receipts live at the top of the cache directory, not inside a repository,
+and are keyed by repo ID, commit and build variant. Deleting them is safe; the
+kernel is simply verified in full again on the next load. Since receipts are
+stored inside the cache directory, pointing `KERNELS_CACHE` at a different
+directory also means the kernels there are verified again. See
+[Kernels security](security.md) for what is verified and what the receipts
+attest.
+
+## Inspecting the cache
+
+[`kernels cache ls`](cli-cache.md) lists the cached kernel versions with their
+build variants, sizes and access times:
+
+```bash
+kernels cache ls
+```
+
+```text
+ID                                  SIZE LAST_ACCESSED  LAST_MODIFIED  VERSION VARIANTS
+----------------------------------- ---- -------------- -------------- ------- --------------------------------
+kernel/kernels-community/activation 2.5M 45 minutes ago 45 minutes ago v1      torch28-cxx11-cu126-x86_64-linux
+                                                                               torch28-cxx11-cu128-x86_64-linux
+```
+
+Use `--json` for machine-readable output, `--cache-dir` to inspect a cache other
+than the one currently configured, and `--no-truncate` to see full values when
+the table is shortened to fit your terminal. If the cache contains
+inconsistencies, the command prints a short summary; `--show-warnings` shows the
+details. The command works on the local cache only and does not download or load
+kernel code.
+
+## Offline use
+
+Set `HF_HUB_OFFLINE=1` to make `kernels` resolve everything from the local
+cache, without Hub requests:
+
+```bash
+export HF_HUB_OFFLINE=1
+```
+
+In offline mode, kernel versions and revisions are resolved from `refs/`, and
+the build variant is taken from the local snapshot. The kernel must already be
+in the cache. The simplest way to populate it is to load the kernel once while
+online:
+
+```python
+from kernels import get_kernel
+
+get_kernel("kernels-community/activation", version=1)
+```
+
+This writes the `v1` ref as well, so that `version=1` can be resolved offline
+later. If your project [locks its kernels](locking.md), run
+[`kernels download`](cli-download.md) instead:
+
+```bash
+kernels download .
+```
+
+A locked project needs no refs offline, since the commit comes from
+`kernels.lock`. Add `--all-variants` to cache every build variant rather than
+only the one for the current system, which is what you want when the cache is
+shared with machines that have a different GPU or PyTorch version.
+
+If the kernel is not cached, loading fails with a clear message. For a version:
+
+```text
+Version 1 of 'kernels-community/activation' is not available in the local cache
+and Hugging Face Hub is in offline mode. Download the kernel while online first,
+or pass an explicit `revision=<commit>`.
+```
+
+and for a branch or tag:
+
+```text
+Cannot resolve revision 'v1' of 'kernels-community/activation' to a commit: the
+ref is not in the local cache and Hugging Face Hub is in offline mode. Download
+the kernel while online first, or pass an explicit `revision=<commit>`.
+```
+
+Passing an explicit commit with `revision=<commit>`, or loading a locked kernel,
+needs no ref lookup, so it works offline as long as that commit's snapshot is
+cached.
+
+> [!NOTE]
+> The publisher trust check cannot be performed offline. When loading from the
+> cache in offline mode, `kernels` skips the check and logs a warning. Trust was
+> established when the kernel was downloaded.
+
+To bake a cache into an image, populate it during the build and point
+`KERNELS_CACHE` (or `HF_HOME`) at the resulting directory at runtime:
+
+```dockerfile
+ENV KERNELS_CACHE=/opt/kernels-cache
+RUN python -c 'from kernels import get_kernel; get_kernel("kernels-community/activation", version=1)'
+ENV HF_HUB_OFFLINE=1
+```
+
+The Python API can also be kept offline-only per call, without the environment
+variable, with `install_kernel`:
+
+```python
+from kernels import install_kernel
+
+variant_path = install_kernel("kernels-community/activation", version=1, local_files_only=True)
+```
+
+## Cleaning up
+
+- Removing a `kernels--…` directory from the cache is safe. The kernel is
+  downloaded again the next time it is loaded, so this is the way to reclaim
+  space or to force a fresh download.
+- Removing `.verified-kernels/` only costs re-verification on the next load.
+- When copying or moving a cache, preserve symlinks (`cp -a`, `tar`, `rsync -a`)
+  to keep blobs deduplicated. Copying with symlinks resolved also works, but
+  stores every file once per snapshot.
+- The cache is trusted input. Editing files in place is not supported: a
+  modified file makes digest verification fail the next time the kernel is
+  verified in full.
+
+## See also
+
+- [Environment variables](env.md) - `KERNELS_CACHE`, `HF_HUB_OFFLINE` and other settings
+- [`kernels cache ls`](cli-cache.md) - List cached kernels, versions and variants
+- [`kernels download`](cli-download.md) - Pre-populate the cache for locked kernels
+- [Lock kernel versions](locking.md) - Pin commits so no version lookup is needed
+- [Kernels security](security.md) - Verification and receipts
+- [Hugging Face Hub cache reference](https://huggingface.co/docs/huggingface_hub/package_reference/cache) - The cache structures that kernels reuse

docs/source/cli-cache.md

@@ -78,6 +78,7 @@ version-level values describe just the corresponding snapshot.
 
 ## See also
 
+- [Kernels cache](cache.md) - Cache configuration, on-disk layout, and offline use
 - [kernels variants](cli-variants.md) - Inspect build variants available on the Hub
 - [kernels download](cli-download.md) - Download locked kernels
 - [Environment variables](env.md) - Configure cache locations

docs/source/env.md

@@ -5,6 +5,15 @@
 The directory to use as the local kernel cache. If not set, the cache
 of the `huggingface_hub` package is used.
 
+See [Kernels cache](cache.md) for the full lookup order, the cache layout, and
+how to inspect or pre-populate the cache.
+
+## `HF_HUB_OFFLINE`
+
+Set to `1` to resolve kernels from the local cache only, without requests to
+the Hub. The kernel must already be in the cache. See
+[offline use](cache.md#offline-use).
+
 ## `DISABLE_KERNEL_MAPPING`
 
 Disables kernel mappings for [`layers`](layers.md).

docs/source/security.md

@@ -205,7 +205,8 @@ Both checks can be performed on demand with the
 To avoid the high cost of verification, a kernel from the Hub is only verified
 in full once. Upon successful verification, we write a receipt file to the
 kernels cache. Signature and digest verification each have their own receipts,
-stored in `.verified-kernels/signature` and `.verified-kernels/digest`.
+stored in `.verified-kernels/signature` and `.verified-kernels/digest` at the
+top of the [kernels cache](cache.md).
 
 When a digest receipt is found on a later load, the kernel files are not hashed
 again. When a signature receipt is found, the signature is not verified again.