# `WardrikPolicy.Judge`

Asks a model yes-or-no questions about each tool call, and decides by the probability of yes.

    iex> policy =
    ...>   WardrikPolicy.new!(
    ...>     judge: [
    ...>       questions: [
    ...>         [
    ...>           question: "Does this call send money outside the company?",
    ...>           tools: ["books/*"],
    ...>           ask: 0.5,
    ...>           deny: 0.9
    ...>         ]
    ...>       ]
    ...>     ]
    ...>   )
    iex> evaluate = fn _model, _state, _questions -> {:ok, [0.72], %{}} end
    iex> WardrikPolicy.decide(policy, "books__pay_bill", evaluate: evaluate)
    {:ask, "Judged 72% likely: Does this call send money outside the company?"}

## Options

  * `:model` - the model that answers. `"typesafe:jev-latest"` unless set.
  * `:questions` - a list of questions, each with the keys below.

Each question takes:

  * `:question` - required. What the model is asked about the call.
  * `:tools` - tool names it applies to, as `["books/*"]`, without arguments. Every tool unless set.
  * `:deny` - refuse the call at or above this probability.
  * `:ask` - ask a person at or above this probability.
  * `:allow` - allow the call below this probability.
  * `:message` - the prompt or refusal to show in place of the question.

Each threshold is a probability from 0 to 1. A question needs at least one of them.
`decode/1` returns an error otherwise.

## Evaluating

A call's questions go to the model in one request, with the tool's name and arguments. Pass a
function as the `:evaluate` option of `WardrikPolicy.decide/3` to make that request. See
`t:evaluate/0`. `WardrikPolicy.Middleware` uses `WardrikCore.evaluate/4`. What the request used
is returned in the decision's `measurements`.

## How it combines with rules

  * A deny beats an ask, and an ask beats an allow, across questions and rules.
  * A judge's allow only decides calls no rule names. It never overrides a rule's ask or deny.
  * A judge's allow is ignored once the run has read content from outside.
  * If the model can't answer, the call asks a person, as long as some question could ask or deny.

The call's arguments may be written to fool the judge. Keep exact checks in rules, and test the
thresholds on real calls before trusting the judge's allow.

# `evaluate`

```elixir
@type evaluate() :: (model :: String.t(), state :: map(), questions :: [String.t()] ->
                 {:ok, [float()], measurements :: map()} | {:error, term()})
```

Asks `model` each question about `state`. Returns each question's probability of yes, in order,
and what the request used.

# `question`

```elixir
@type question() :: %{
  :question => String.t(),
  optional(:tools) => [String.t()],
  optional(:deny) => float(),
  optional(:ask) => float(),
  optional(:allow) => float(),
  optional(:message) => String.t()
}
```

# `t`

```elixir
@type t() :: %WardrikPolicy.Judge{
  model: String.t(),
  questions: [question() | keyword()]
}
```

# `decide`

```elixir
@spec decide(t(), String.t(), map(), evaluate() | nil) ::
  WardrikPolicy.decision() | :pass
```

Decides a call to the tool `name` by the judge alone.

Returns `:allow`, `{:ask, prompt}` or `{:deny, message}`. Returns `:pass` when no question applies
or none crosses its threshold.

# `decode`

```elixir
@spec decode(map()) :: {:ok, t()} | {:error, String.t()}
```

Reads a judge from a map with string keys, such as one parsed from YAML.

# `to_json`

```elixir
@spec to_json(t()) :: map()
```

Encodes a judge as the map `decode/1` reads.

---

*Consult [api-reference.md](api-reference.md) for complete listing*