Jev

Who it is for

Operators adding an external model’s advisory judgment to an existing agent workflow, while keeping the workflow’s risk controls authoritative.

The problem and constraints

An advisory model can offer useful categorization and still make a consequential mistake. A request also needs a clear owner for model selection, deadlines, and budget; otherwise each integration surface can acquire its own behavior.

How it is used

Hooks, MCP tools, and skills call thin adapters around a single client. That client owns the request, model, deadline, and budget. Its output can raise the existing risk level, but it cannot lower it.

If the model is missing or errors, the workflow remains conservative. The integration does not replace the external model with a proprietary one.

The advisory response can raise the existing risk level but never lower it. A missing response or an error creates no lower-risk path.

Open diagram at full size

Read the workflow as text

• Integration surfaces: hooks, MCP tools and skills call thin adapters.
• Single client: owns request, model, deadline, budget. The adapters call this one client.
• The client asks the external model for advice. Response received? That check decides the next step.
• If yes, the response goes to the raise-only mapping: same or higher risk, starting from the existing risk assessment.
• On a missing or error result, stay conservative: no lower-risk path.

The mapping has one invariant: the advisory class can raise the class a person set, never lower it, and a missing answer counts against the change. To test it without a live model, on October 2, 2026, I ran the unchanged code that turns Jev’s answers into a risk class on four synthetic briefs. A local stub served fixed probabilities for six checks, f1 to f6, and the score counts the checks that came out yes. In brief 2 the advisory class was STANDARD at 2/6, below the person’s CRITICAL, and the person’s class stood. In brief 3 the changed file was an auth guard, a code rule answered f1 without asking the model, and the advisory class came out CRITICAL against the person’s STANDARD, which the lead may use to raise it. In brief 4 the stub answered HTTP 503, every check counted as a yes, and the score went to 6/6 and CRITICAL, so the error path stayed conservative.

$ cd ~/sandbox && TMPDIR=~/sandbox/tmp tsx run-jev-demo.ts; echo "exit=$?"

[1] brief-docs-typo.md  (human line: CAR-CLASS: ROUTINE)
    manifest: docs/guide.md
    asked Jev: f1,f2,f3,f4,f5,f6  ->  reply: f1=0.04 f2=0.03 f3=0.06 f4=0.11 f5=0.07 f6=0.05
    printed:   CAR-CLASS-JEV: 0/6 → ROUTINE (advisory; f4=0.11 f5=0.07; jev-1.13.0)
    brief now: CAR-CLASS: ROUTINE  |  CAR-CLASS-JEV: 0/6 → ROUTINE (advisory; f4=0.11 f5=0.07; jev-1.13.0)

[2] brief-dep-bump.md  (human line: CAR-CLASS: CRITICAL)
    manifest: package.json, package-lock.json
    asked Jev: f1,f2,f3,f4,f5,f6  ->  reply: f1=0.22 f2=0.08 f3=0.12 f4=0.61 f5=0.05 f6=0.47
    printed:   CAR-CLASS-JEV: 2/6 → STANDARD (advisory; f4=0.61 f5=0.05; jev-1.13.0) — below the human line; human stands
    brief now: CAR-CLASS: CRITICAL  |  CAR-CLASS-JEV: 2/6 → STANDARD (advisory; f4=0.61 f5=0.05; jev-1.13.0) — below the human line; human stands

[3] brief-auth-check.md  (human line: CAR-CLASS: STANDARD)
    manifest: src/auth/session-guard.ts, tests/auth/session-guard.test.ts
    asked Jev: f2,f3,f4,f5,f6  ->  reply: f2=0.71 f3=0.18 f4=0.35 f5=0.12 f6=0.09
    printed:   CAR-CLASS-JEV: 2/6 → CRITICAL (advisory; f4=0.35 f5=0.12; jev-1.13.0) — above the human line; the lead may raise, never lower
    brief now: CAR-CLASS: STANDARD  |  CAR-CLASS-JEV: 2/6 → CRITICAL (advisory; f4=0.35 f5=0.12; jev-1.13.0) — above the human line; the lead may raise, never lower

[4] brief-docs-typo-outage.md  (human line: CAR-CLASS: ROUTINE)
    manifest: docs/guide.md
    asked Jev: f1,f2,f3,f4,f5,f6  ->  reply: HTTP 503
    printed:   CAR-CLASS-JEV: 6/6 → CRITICAL (advisory; f4=err f5=err; jev-1.13.0) — above the human line; the lead may raise, never lower
    brief now: CAR-CLASS: ROUTINE  |  CAR-CLASS-JEV: 6/6 → CRITICAL (advisory; f4=err f5=err; jev-1.13.0) — above the human line; the lead may raise, never lower

shadow ledger rows: 4; last: {"brief":"briefs/brief-docs-typo-outage.md","human_class":"ROUTINE","jev_class":"CRITICAL","score":6,"filters":{"f1":true,"f2":true,"f3":true,"f4":true,"f5":true,"f6":true},"f4":null,"f5":null,"model":"jev-1.13.0","ms":1,"reason":"http_503"}
exit=0

Why the advisory boundary matters

Why the threshold leans toward higher risk

The decomposed approach turns six yes-or-no answers into a risk class, so the cut-off for each answer decides how often a brief is downgraded. At the default threshold of 0.5, it gave a lower risk class than the label to seven briefs labelled high risk. Moving the threshold to 0.40, so that uncertain answers lean toward the higher class, reduced that to one. That is why the later result uses 0.40. These are the same 40 briefs, from September 2026.

Evaluation evidence

I ran five small trials in September 2026 to see where an external classifier helps and where it fails. The trials sent workflow text to that model, so I scrubbed secrets before any data left the machine and dropped any row that carried something that looked like a secret.

The 80 pairs were selected because their titles looked alike. As a negative control, I also scored 40 randomly paired issues. None scored 0.5 or higher as a duplicate; the highest was 0.07.

A separate triage record detected six labelled-positive cases and added three false positives among 54 labelled-negative cases. These are also comparisons with a labelled evaluation set, not measured performance on new work, and independent human validation of those labels has not been established either.

Two failure modes the trials exposed

One trial asked for the cause of each failure in a set of log rows. In 17 rows the label said the cause was unknown, and the model still named a cause for 6 of them. A multiple-choice answer has to pick something, so every such question needs an explicit unknown or none option.

The trials also showed a contrast between two kinds of input. On short, self-contained text such as issue titles, most answers were decisive: below 0.1 or above 0.9. On long, noisy text such as the opening of a task brief, answers tended to fall between 0.3 and 0.7, and answers in that band told me little. Those inputs also came from different tasks, so the trials did not isolate length as the cause.

Decisions and current state

The integration centralizes calls in one client instead of letting hooks, MCP tools, and skills each own model and request policy. The risk mapping is raise-only because the evaluation still contained a critical downgrade.

The record supports an advisory integration and small selected evaluations. It does not establish broad adoption, token savings, production latency, or a general quality improvement.

Learnings

What I learned: better agreement on a small set can coexist with an unacceptable individual risk decision. Aggregate improvement cannot justify removing the boundary.

What changed: the advisory mapping can only raise risk, and missing/error paths stay conservative.

What to test next: preserve raw request/response records, evaluate a larger, independently labelled set of new cases, separate threshold changes from decomposition, and test the raise-only invariant directly.