bug(manifests): Jailbreak rail manifest conflates in-process vs remote, model vs heuristics with conflicting dependencies

[tgasser-nv]

Title

Did you check docs and existing issues?

• I have read all the NeMo-Guardrails documentation
• I have updated the package to the latest version before submitting this issue
• (optional) I have used the develop branch
• I have searched the existing issues of NeMo-Guardrails

Python version

3.13.2

Operating system/version

macOS (Darwin 25.5.0)

NeMo-Guardrails version

0.24.0.dev0 (develop)

---

Describe the bug

nemoguardrails/library/jailbreak_detection/rail.py declares:

optional_dependencies=("scikit-learn", "torch"),

Three things are wrong with that list, and a fourth problem means no corrected list can be right.

1. scikit-learn is declared but never imported.
JailbreakClassifier loads the random forest through ONNX Runtime, not scikit-learn:

# nemoguardrails/library/jailbreak_detection/model_based/models.py:56
from onnxruntime import InferenceSession
...
self.classifier = InferenceSession(random_forest_path, providers=["CPUExecutionProvider"])

The sklearn-onnx URL on line 59 is a comment describing the format the model was exported from.
scikit-learn appears nowhere in the library except this declaration, that comment, and one log message.
onnxruntime, which is genuinely required, is a core dependency (pyproject.toml:27-28) so it happens to be present regardless.

2. transformers is required but was not declared.
Both in-process paths import it, one at module scope:

• heuristics/checks.py:19 — from transformers import GPT2LMHeadModel, GPT2TokenizerFast
• model_based/models.py:27 — from transformers import AutoModel, AutoTokenizer

The package's own requirements.txt lists transformers>=5.3.0.

3. The user-facing remediation message is wrong.
nemoguardrails/library/jailbreak_detection/actions.py:158, logged when the in-process import fails:

Failed to import required dependencies for local model. Install scikit-learn and torch, or use NIM-based approach

Someone following this installs a package that is not used, does not install transformers, and hits the same failure again.

4. The structural problem: requirements here are conditional, and the field is not.

RailRequirements.optional_dependencies is one unconditional tuple per manifest.
The jailbreak packages are needed only on the in-process path, and the two surfaces do not need the same things:

surface	in-process	remote
jailbreak detection heuristics	torch, transformers	nothing (server_endpoint)
jailbreak detection model	torch, transformers, onnxruntime	nothing (server_endpoint or nim_base_url)

The two surfaces also disagree about which config field means "remote": the model rail treats nim_base_url as remote, heuristics has its own server and ignores it.

So even a corrected tuple stays wrong in one direction or the other — declare the packages and every server-backed config is refused, omit them and an in-process config passes validation then fails on the first request.

Steps To Reproduce

Wrong declaration (1 and 2):

uv run --locked python - <<'PY'
import inspect
from nemoguardrails.manifests import all_rail_manifests
print(all_rail_manifests()["jailbreak_detection"].requirements.optional_dependencies)
# ('scikit-learn', 'torch')  -- names sklearn, omits transformers
PY

grep -rn "import transformers\|from transformers" nemoguardrails/library/jailbreak_detection/
grep -rn "sklearn\|scikit" nemoguardrails/library/jailbreak_detection/ --include="*.py"

Conditional requirements (4), on a build without torch/transformers installed:

# config.yml -- a working NIM deployment
models:
  - type: main
    engine: nim
    model: meta/llama-3.3-70b-instruct
rails:
  input:
    flows:
      - jailbreak detection model
  config:
    jailbreak_detection:
      nim_base_url: https://ai.api.nvidia.com
      nim_server_endpoint: /v1/security/nvidia/nemoguard-jailbreak-detect

This config needs none of the declared packages. Any consumer that enforces optional_dependencies must special-case the rail to avoid rejecting it.

Expected Behavior

A manifest states what a rail needs to run, accurately, and a consumer can act on it without per-rail knowledge.

For jailbreak that means the requirements are expressible per backend (and per surface, since the two surfaces select backends differently) rather than as one tuple per manifest.

Actual Behavior

optional_dependencies names one package the rail never imports and, until recently, omitted one it imports at module scope.
Because the tuple cannot be qualified, a consumer enforcing it has to choose between rejecting working remote configs and letting in-process configs fail at request time.

---

Impact on IORails (context, not part of the bug)

nemoguardrails/guardrails/compiled_rail.py refuses at compile time when a manifest's declared distributions are missing, so a missing extra is reported once as a configuration error rather than reaching a request, where the fail-closed envelope renders it as a content block.

Since jailbreak's requirements cannot be qualified, PR #2264:

• adds transformers to the declaration, so the message is at least accurate
• keeps a config-reading backend check for jailbreak detection model (remote on server_endpoint or nim_base_url)
• blocklists jailbreak detection heuristics in _unsupported_rail_reason, because it shares a config section with the model rail but reads a different field, so no manifest-wide answer is right for both

That blocklist entry exists only because the manifest cannot express this, and should be removed when it can.

Proposed fix

Independently correctable, and worth doing regardless:

1. Drop scikit-learn, add onnxruntime (or drop it too, as a core dependency), add transformers
2. Fix the remediation message at actions.py:158 to name what is actually needed

Needs design (tracked in #2279):

3. Let requirements be declared per backend, so a rail with an in-process and a remote path can state both. Two shapes were considered while working on feat(iorails): Support blocking rails #2264:
   • a remote_when declaration on RailSurface naming the config keys that select a remote backend — covers jailbreak, does not cover hf_classifier, whose selector is an enum comparison nested under a per-classifier key
   • promoting the backend to a surface parameter (jailbreak detection model $backend=nim, mirroring $model= and $classifier=), so the choice is statically visible in the flow string and requirements can be declared per parameter value. This generalises to hf_classifier's engine with one mechanism and needs no new predicate machinery.

A cheaper interim step that makes the declaration honest without new schema: a conformance test asserting every declared distribution is imported somewhere in its library package. That alone would have caught (1).

Related

• feat(iorails): Support blocking rails #2264 — IORails blocking rails; adds the compile-time dependency check that surfaced this
• Rail Manifest required updates #2279 — manifest cannot express what IORails needs to know about a rail