xnano.ai.testing
xnano.ai.testing
¶
xnano.ai.testing
Models that answer from a script, so programs and tests that use xnano.ai run offline and give
the same answers every time.
ScriptedDecisionModel is a real pydantic-ai decision
model whose answers come from a mapping, so the questions, probabilities and margins a task model
works with are the ones a real decision backend would give.
scripted_model is a language model that answers each
request with the next step of a script.
replay runs a file of cases through a
ChoiceModel once, and reports how often it is right at each
threshold of probability, how well its probabilities are calibrated, and whether the order of
the options changes its answers: how a threshold for a program is chosen.
Classes:
-
ScriptedDecisionModel–A decision model that answers every question from a script, with no network.
-
ScriptedModel–What
scripted_modelreturns: the model, and a record. -
ReplayCase–One case of a replay: the answer expected, and the one the model gave.
-
Coverage–How a threshold of probability would work: how many cases it answers, how many correctly.
-
ReplayReport–What a replay found: every case, and the figures that choose a threshold.
Functions:
-
scripted_model–A language model that answers each request with the next step of a script.
-
replay_async–Run cases through a choice model once, and report how it did, with
await. -
replay–Run cases through a choice model once, and report how it did, waiting for it.
ScriptedAnswer
module-attribute
¶
One scripted answer: for a yes/no, True, False or the probability of a yes; for a
pick-one, the label or a mapping from label to probability; for a rubric, the level or a
mapping from level to probability.
AnswerFunction
module-attribute
¶
AnswerFunction: TypeAlias = Callable[[str, DecisionQuestion, DecisionRequest], ScriptedAnswer]
A function from the name of a question, the question and the request to its answer.
ScriptedDecisionModel
dataclass
¶
ScriptedDecisionModel(answers: Mapping[str, ScriptedAnswer] | AnswerFunction | None = None, *, model_name: str = 'scripted')
Bases: DecisionModel[None]
A decision model that answers every question from a script, with no network.
answers maps the name of a question to its answer: 'response' for a single
pick-one, a field name such as 'answer' or 'level' otherwise. A question with no
scripted answer gets a yes with probability 0.75, the first option, or level 0.
Parameters:
-
answers(Mapping[str, ScriptedAnswer] | AnswerFunction | None, default:None) –The answers by question name, or a function that returns the answer for a question; see
ScriptedAnswer. -
model_name(str, default:'scripted') –The name the model reports.
Attributes:
-
answers(Mapping[str, ScriptedAnswer] | AnswerFunction) –The script.
-
requests(list[DecisionRequest]) –Every request the model was sent, oldest first.
-
last_state(object) –What the latest request judged, parsed from JSON when it is JSON.
Examples:
from xnano.ai.models.decisions import ChoiceModel
from xnano.ai.testing import ScriptedDecisionModel
model = ScriptedDecisionModel({'response': {'left': 0.6, 'right': 0.4}})
decision = ChoiceModel(model).choose({'x': 3}, ['left', 'right'])
assert decision.choice == 'left'
assert model.last_state == {'x': 3}
Create the model with its script.
Methods:
-
request–Answer a request; a local model, so it is allowed when model requests are not.
-
decide–Answer every question of the request from the script.
Source code in xnano/ai/testing.py
last_state
property
¶
last_state: object
What the latest request judged, parsed from JSON when it is JSON, or None.
request
async
¶
Answer a request; a local model, so it is allowed when model requests are not.
decide
async
¶
Answer every question of the request from the script.
Source code in xnano/ai/testing.py
async def decide(
self, request: DecisionRequest, model_settings: DecisionModelSettings
) -> DecisionResponse:
"""Answer every question of the request from the script."""
self.requests.append(request)
answers: dict[str, DecisionAnswer] = {}
for name, question in request.questions.items():
value = self._scripted(name, question, request)
if isinstance(question, NoulQuestion):
answers[name] = _yes_or_no_answer(value)
elif isinstance(question, ChoiceQuestion):
answers[name] = _choice_answer(question, value)
elif isinstance(question, ScoreQuestion):
answers[name] = _score_answer(question, value)
return DecisionResponse(answers=answers, model_name=self._name)
ScriptedModel
dataclass
¶
ScriptedModel(model: FunctionModel, calls: list[AgentInfo] = list(), messages: list[list[ModelMessage]] = list())
What scripted_model returns: the model, and a record.
Attributes:
-
model(FunctionModel) –The pydantic-ai
FunctionModelto give a task model. -
calls(list[AgentInfo]) –What the agent offered at each request: its output tools and settings.
-
messages(list[list[ModelMessage]]) –The messages of each request, oldest first.
ReplayCase
dataclass
¶
One case of a replay: the answer expected, and the one the model gave.
Attributes:
-
expected(str) –The option the case expects.
-
choice(str) –The option the model picked.
-
probability(float | None) –The probability the model gave its pick, or
Noneon a language model. -
reordered_choice(str | None) –The option it picked with the options in the reverse order, or
Nonewhen the order was not checked. -
is_correct(bool) –Whether the pick is the option expected.
Coverage
dataclass
¶
How a threshold of probability would work: how many cases it answers, how many correctly.
Attributes:
-
threshold(float) –The least probability a pick needs to be acted on.
-
coverage(float) –The share of the cases whose pick reaches the threshold.
-
accuracy(float | None) –The share of those that are correct, or
Nonewhen none reach it.
ReplayReport
dataclass
¶
ReplayReport(cases: tuple[ReplayCase, ...])
What a replay found: every case, and the figures that choose a threshold.
Attributes:
-
cases(tuple[ReplayCase, ...]) –Every case, in the order of the file.
-
accuracy(float) –The share of the cases the model got right.
-
majority_baseline(float) –The accuracy of always picking the option most cases expect: the figure the model has to beat.
-
calibration_error(float | None) –The expected calibration error over ten bins of probability: how far the probabilities are from how often the picks are right, from
0.0(the same) up.Noneon a language model, which gives no probabilities. -
order_sensitivity(float | None) –The share of the cases whose pick changed when the options were given in the reverse order, or
Nonewhen that was not checked.
Examples:
from xnano.ai.models.decisions import ChoiceModel
from xnano.ai.testing import ScriptedDecisionModel, replay
model = ScriptedDecisionModel({'response': {'left': 0.8, 'right': 0.2}})
cases = [
{'evidence': {'wall': 'right'}, 'options': ['left', 'right'], 'expected': 'left'},
{'evidence': {'wall': 'left'}, 'options': ['left', 'right'], 'expected': 'right'},
]
report = replay(ChoiceModel(model), cases, should_check_order=False)
assert report.accuracy == 0.5
assert report.at_threshold(0.7).coverage == 1.0
Methods:
-
at_threshold–How acting only on picks of at least
thresholdwould have worked.
majority_baseline
property
¶
majority_baseline: float
The accuracy of always picking the option most cases expect.
calibration_error
property
¶
calibration_error: float | None
The expected calibration error over ten bins of probability, or None.
order_sensitivity
property
¶
order_sensitivity: float | None
The share of the cases whose pick changed with the options reversed, or None.
at_threshold
¶
How acting only on picks of at least threshold would have worked.
Parameters:
-
threshold(float) –The least probability, from
0.0to1.0.
Raises:
-
ValueError–thresholdis outside0.0to1.0.
Source code in xnano/ai/testing.py
def at_threshold(self, threshold: float) -> Coverage:
"""
How acting only on picks of at least ``threshold`` would have worked.
Args:
threshold: The least probability, from ``0.0`` to ``1.0``.
Raises:
ValueError: ``threshold`` is outside ``0.0`` to ``1.0``.
"""
if isinstance(threshold, bool) or not 0.0 <= threshold <= 1.0:
raise ValueError(f'threshold must be from 0.0 to 1.0, but was {threshold!r}.')
reached = [
case
for case in self.cases
if case.probability is not None and case.probability >= threshold
]
return Coverage(
threshold=threshold,
coverage=len(reached) / len(self.cases) if self.cases else 0.0,
accuracy=_share(case.is_correct for case in reached) if reached else None,
)
scripted_model
¶
scripted_model(*steps: object, repeat_last: bool = False) -> ScriptedModel
A language model that answers each request with the next step of a script.
A step is text, a mapping of the arguments of the output (such as {'response':
'left'}), a (tool name, arguments) pair, a pydantic-ai ModelResponse, or a function
of the messages and the AgentInfo that returns one of these.
Parameters:
-
steps(object, default:()) –The answers, in order.
-
repeat_last(bool, default:False) –Whether the last step answers every request after it.
Returns:
-
ScriptedModel–The
ScriptedModel: give itsmodelto a task -
ScriptedModel–model.
Examples:
from xnano.ai.models.text import TextModel
from xnano.ai.testing import scripted_model
script = scripted_model('The door creaks open.')
writer = TextModel(script.model)
assert writer.generate('Describe the door.') == 'The door creaks open.'
Source code in xnano/ai/testing.py
def scripted_model(*steps: object, repeat_last: bool = False) -> ScriptedModel:
"""
A language model that answers each request with the next step of a script.
A step is text, a mapping of the arguments of the output (such as ``{'response':
'left'}``), a ``(tool name, arguments)`` pair, a pydantic-ai ``ModelResponse``, or a function
of the messages and the ``AgentInfo`` that returns one of these.
Args:
steps: The answers, in order.
repeat_last: Whether the last step answers every request after it.
Returns:
The [`ScriptedModel`][xnano.ai.testing.ScriptedModel]: give its ``model`` to a task
model.
Examples:
```python
from xnano.ai.models.text import TextModel
from xnano.ai.testing import scripted_model
script = scripted_model('The door creaks open.')
writer = TextModel(script.model)
assert writer.generate('Describe the door.') == 'The door creaks open.'
```
"""
queue = list(steps)
record = ScriptedModel(
model=FunctionModel(lambda messages, info: _next(messages, info))
)
def _next(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
record.calls.append(info)
record.messages.append(list(messages))
if not queue:
raise AssertionError('The scripted model has no step left for this request.')
step = queue[0] if repeat_last and len(queue) == 1 else queue.pop(0)
if callable(step) and not isinstance(step, (BaseModel, type)):
step = step(messages, info)
return _step_response(step, info)
async def _stream(
messages: list[ModelMessage], info: AgentInfo
) -> AsyncIterator[str | DeltaToolCalls]:
response = _next(messages, info)
for part in response.parts:
if isinstance(part, TextPart):
# Word by word, the way a model streams its text.
for piece in re.findall(r'\S+\s*|\s+', part.content):
yield piece
elif isinstance(part, ToolCallPart):
yield {
0: DeltaToolCall(
name=part.tool_name, json_args=part.args_as_json_str()
)
}
record.model = FunctionModel(_next, stream_function=_stream, model_name='scripted')
return record
replay_async
¶
replay_async(model: ChoiceModel, cases: str | PathLike[str] | Sequence[Mapping[str, Any]], *, should_check_order: bool = True) -> Coroutine[Any, Any, ReplayReport]
Run cases through a choice model once, and report how it did, with await.
Each case is a mapping, or a line of a JSONL file, with evidence (what the model
judges), options (a list of labels, or a mapping from label to what it means), the
expected label and, when it differs from the model's, a question. The model is
asked once per case, and once more with the options in the reverse order unless
should_check_order is false; every figure of the report comes from those answers, with
no new requests.
Parameters:
-
model(ChoiceModel) –The
ChoiceModelto replay. -
cases(str | PathLike[str] | Sequence[Mapping[str, Any]]) –The cases, or the path of a JSONL file of them.
-
should_check_order(bool, default:True) –Whether to ask each case again with the options reversed.
Returns:
-
Coroutine[Any, Any, ReplayReport]–A coroutine that gives the
ReplayReport.
Raises:
-
ValueError–A case has no
evidence,optionsorexpected, or expects a label that is not one of its options.
Source code in xnano/ai/testing.py
def replay_async(
model: ChoiceModel,
cases: str | os.PathLike[str] | Sequence[Mapping[str, Any]],
*,
should_check_order: bool = True,
) -> Coroutine[Any, Any, ReplayReport]:
"""
Run cases through a choice model once, and report how it did, with ``await``.
Each case is a mapping, or a line of a JSONL file, with ``evidence`` (what the model
judges), ``options`` (a list of labels, or a mapping from label to what it means), the
``expected`` label and, when it differs from the model's, a ``question``. The model is
asked once per case, and once more with the options in the reverse order unless
``should_check_order`` is false; every figure of the report comes from those answers, with
no new requests.
Args:
model: The [`ChoiceModel`][xnano.ai.models.decisions.ChoiceModel] to replay.
cases: The cases, or the path of a JSONL file of them.
should_check_order: Whether to ask each case again with the options reversed.
Returns:
A coroutine that gives the [`ReplayReport`][xnano.ai.testing.ReplayReport].
Raises:
ValueError: A case has no ``evidence``, ``options`` or ``expected``, or expects a
label that is not one of its options.
"""
loaded = _load_cases(cases)
return _replay(model, loaded, should_check_order)
replay
¶
replay(model: ChoiceModel, cases: str | PathLike[str] | Sequence[Mapping[str, Any]], *, should_check_order: bool = True) -> ReplayReport
Run cases through a choice model once, and report how it did, waiting for it.
The same as replay_async.
Source code in xnano/ai/testing.py
def replay(
model: ChoiceModel,
cases: str | os.PathLike[str] | Sequence[Mapping[str, Any]],
*,
should_check_order: bool = True,
) -> ReplayReport:
"""
Run cases through a choice model once, and report how it did, waiting for it.
The same as [`replay_async`][xnano.ai.testing.replay_async].
"""
from xnano.ai.models import _run_blocking
return _run_blocking(
replay_async(model, cases, should_check_order=should_check_order), 'replay'
)