llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev)

[ngxson]

Overview

Support 5 decision models: laya, julia-1, lev, openjev (+vision), kev

Pre-converted GGUF:

llama-server -hf ggml-org/OpenJev-GGUF
llama-server -hf ggml-org/lev-GGUF
llama-server -hf ggml-org/Laya-GGUF
llama-server -hf ggml-org/Julia-1-GGUF
llama-server -hf ggml-org/Kev-4B-GGUF

Additional information

"Decision models" are just fancy wrappers around traditional embedding models (BERT/Qwen/etc), so changes to support them are kept to be as small and as self-contained as possible.

Since the underlay infra is just existing embedding models, changes to libllama are minimal. Main part is server_decision_context that switches the input/output handling based on {arch}.decision.type gguf metadata.

Recap changes per-component:

• Conversion: handle conversion + new tensors/metadata + chat template injection
• libllama: add decision head on graph, add some getters API
• Server: add /v1/systemone API, add the main server_decision_context handling logic, add tests (using a slice of laya model)

TODO:

• maybe support https://huggingface.co/Cloudflare/clef , just out 2 hours ago at the time writing this planned as a follow-up
• add docs (+ dev docs for those who will add future model support)
• support parallel shared prompt prefix (only for causal models)
• support vision input

Testing

Table represent result in format: reference → llama.cpp

input json

{
  "state": {
    "message": "Hi, I was charged twice for my order #4471 and I want a refund.",
    "plan": "pro",
    "order": {
      "id": 4471,
      "items": ["phone case", "charger"]
    }
  },
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "What does the customer want?",
      "criteria": {
        "refund": "wants money back",
        "cancel": "wants to cancel an order",
        "track": "wants to know where an order is",
        "other": "anything else"
      }
    },
    "urgent": {
      "type": "noul",
      "instructions": "Does this need a human within the hour?"
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["calm", "mildly annoyed", "annoyed", "angry"]
    },
    "refund": {
      "type": "noul",
      "instructions": "Is a refund requested?",
      "criteria": {
        "true": "money back is asked",
        "false": "no money back is asked"
      }
    },
    "team": {
      "type": "choice",
      "instructions": "Which team?",
      "criteria": {
        "billing": null,
        "shipping": null,
        "technical": null,
        "sales": null,
        "legal": null,
        "returns": null,
        "fraud": null,
        "accounts": null,
        "retention": null,
        "other": null
      }
    }
  }
}

question	laya	julia-1	lev	openjev	kev-4b
intent (top option, refund)	0.9871 → 0.9870	0.9899 → 0.9910	0.8159 → 0.8209	0.9994 → 0.9994	0.6537 → 0.6550
urgent (noul)	0.1013 → 0.1016	0.6447 → 0.6232	0.4680 → 0.4681	0.2505 → 0.2507	0.7051 → 0.7045
frustration (top level)	0.3492 → 0.3492	0.6749 → 0.6895	0.2627 → 0.2633	0.5295 → 0.5295	0.4406 → 0.4407
refund (noul)	0.8916 → 0.8916	0.5608 → 0.5636	0.9503 → 0.9505	0.9901 → 0.9901	0.9472 → 0.9474
team (top option)	0.9666 → 0.9664	not run	0.3815 → 0.3787	0.9874 → 0.9874	0.4598 → 0.4599
worst diff, any probability	8.4e-4	2.2e-2	5.2e-3	1.8e-4	1.3e-3
input tokens	414 = 414	not reported by ref	576 vs 1194	619 = 619	406 = 406

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: most of the code is AI-written, I own the design

[ggerganov]

Is this correct "lev" instead of "kev" here?

[ngxson]

it's expected, they are quite similar so I grouped them into one file

[ggerganov]

I did some testing with OpenJEV and Laya and seems to work correct. Nice work!

[CISC]

Are we sure this does not negatively affect any models?

[ngxson]

it should only affect models with load_hparams_guess, i.e. models that had config.json at all

now the load_hparams_guess expanded to models that already had config.json, but we want to use a derived class for conversion

btw, now having this, we should probably remove --mistral-format or at least somehow make these mistral-related flags more agnostic

[CISC]

The models missing config.json was what concerned me, but I'm not sure which those are.

[ngxson]

only pockettts.py currently using it, which should be fine

[espetro]

Hi @ngxson, thanks for bringing TypeSafe API parity upstream. The parity table methodology is quite helpful.

I've been testing Kev conversions independently (see fork in case it helps) and have quantized 0.8B/4B/9B GGUFs with embedded decision heads at espetro/kev-4b-demo-gguf (also 0.8b, 9b), produced with an end-to-end probe harness that measures argmax flips and probability drift vs the Python F16 reference. Couple of things that could help out:

1. Footgun worth a line in the conversion docs: merging a decision head into an already-quantized base GGUF (tested with an unrelated Unsloth quant) flipped argmax on 10/17 questions, max |dp| 0.875. Heads need to be merged pre-quantization.
2. If you want extra parity targets: across a 17-question probe suite, our imatrix quants show 0-1 argmax flips, max |dp| 0.051-0.140 (we focused on quantized-vs-F16-reference; different baseline than your F16 parity table, so not directly comparable).

One more thing: KevModel subclasses Qwen3_5TextModel, and the date_facts preprocessing is marked TODO - so Kev 1.0's Qwen3.8-based 27B model needs both a new arch and date_facts support. Is that on the roadmap? Happy to test.

[ngxson]

@espetro I don't quite get what you are talking about: merging LoRA to base is done at conversion time, on pytorch, there is no quantization there. if you are referring to your fork, that's completely unrelated to this PR

date_facts is intentionally left out for simplicity. I don't plan to support it, it's just quite fragile: the date matching using regex, that won't work well on multi-language setup

[kroaton]

Will this support https://huggingface.co/Cloudflare/clef? Seems to be the best model of this class released so far.

[paschembri]

How to use decision models, aka system one endpoint, with the llama.cpp embedded model router:

Adding just the info for anyone wondering how to use it with the model router (as I wondered 10 min ago):

just add the "model": "< model name >" in the json, same level as "state".