Skip to content

xnano.ai.testing

xnano.ai.testing

xnano.ai.testing

Models that answer from a script, so programs and tests that use xnano.ai run offline and give the same answers every time.

ScriptedDecisionModel is a real pydantic-ai decision model whose answers come from a mapping, so the questions, probabilities and margins a task model works with are the ones a real decision backend would give. scripted_model is a language model that answers each request with the next step of a script.

replay runs a file of cases through a ChoiceModel once, and reports how often it is right at each threshold of probability, how well its probabilities are calibrated, and whether the order of the options changes its answers: how a threshold for a program is chosen.

Classes:

  • ScriptedDecisionModel –

    A decision model that answers every question from a script, with no network.

  • ScriptedModel –

    What scripted_model returns: the model, and a record.

  • ReplayCase –

    One case of a replay: the answer expected, and the one the model gave.

  • Coverage –

    How a threshold of probability would work: how many cases it answers, how many correctly.

  • ReplayReport –

    What a replay found: every case, and the figures that choose a threshold.

Functions:

  • scripted_model –

    A language model that answers each request with the next step of a script.

  • replay_async –

    Run cases through a choice model once, and report how it did, with await.

  • replay –

    Run cases through a choice model once, and report how it did, waiting for it.

ScriptedAnswer module-attribute

ScriptedAnswer: TypeAlias = object

One scripted answer: for a yes/no, True, False or the probability of a yes; for a pick-one, the label or a mapping from label to probability; for a rubric, the level or a mapping from level to probability.

AnswerFunction module-attribute

AnswerFunction: TypeAlias = Callable[[str, DecisionQuestion, DecisionRequest], ScriptedAnswer]

A function from the name of a question, the question and the request to its answer.

ScriptedDecisionModel dataclass

ScriptedDecisionModel(answers: Mapping[str, ScriptedAnswer] | AnswerFunction | None = None, *, model_name: str = 'scripted')

Bases: DecisionModel[None]

A decision model that answers every question from a script, with no network.

answers maps the name of a question to its answer: 'response' for a single pick-one, a field name such as 'answer' or 'level' otherwise. A question with no scripted answer gets a yes with probability 0.75, the first option, or level 0.

Parameters:

Attributes:

Examples:

from xnano.ai.models.decisions import ChoiceModel
from xnano.ai.testing import ScriptedDecisionModel

model = ScriptedDecisionModel({'response': {'left': 0.6, 'right': 0.4}})
decision = ChoiceModel(model).choose({'x': 3}, ['left', 'right'])
assert decision.choice == 'left'
assert model.last_state == {'x': 3}

Create the model with its script.

Methods:

  • request –

    Answer a request; a local model, so it is allowed when model requests are not.

  • decide –

    Answer every question of the request from the script.

Source code in xnano/ai/testing.py
def __init__(
    self,
    answers: Mapping[str, ScriptedAnswer] | AnswerFunction | None = None,
    *,
    model_name: str = 'scripted',
) -> None:
    """Create the model with its script."""
    super().__init__()
    self.answers = {} if answers is None else answers
    self.requests = []
    self._name = model_name

model_name property

model_name: str

The name the model reports.

system property

system: str

The provider the model reports: 'scripted'.

last_state property

last_state: object

What the latest request judged, parsed from JSON when it is JSON, or None.

request async

request(*arguments: Any, **keywords: Any) -> ModelResponse

Answer a request; a local model, so it is allowed when model requests are not.

Source code in xnano/ai/testing.py
async def request(self, *arguments: Any, **keywords: Any) -> ModelResponse:
    """Answer a request; a local model, so it is allowed when model requests are not."""
    with override_allow_model_requests(True):
        return await super().request(*arguments, **keywords)

decide async

decide(request: DecisionRequest, model_settings: DecisionModelSettings) -> DecisionResponse

Answer every question of the request from the script.

Source code in xnano/ai/testing.py
async def decide(
    self, request: DecisionRequest, model_settings: DecisionModelSettings
) -> DecisionResponse:
    """Answer every question of the request from the script."""
    self.requests.append(request)
    answers: dict[str, DecisionAnswer] = {}
    for name, question in request.questions.items():
        value = self._scripted(name, question, request)
        if isinstance(question, NoulQuestion):
            answers[name] = _yes_or_no_answer(value)
        elif isinstance(question, ChoiceQuestion):
            answers[name] = _choice_answer(question, value)
        elif isinstance(question, ScoreQuestion):
            answers[name] = _score_answer(question, value)
    return DecisionResponse(answers=answers, model_name=self._name)

ScriptedModel dataclass

ScriptedModel(model: FunctionModel, calls: list[AgentInfo] = list(), messages: list[list[ModelMessage]] = list())

What scripted_model returns: the model, and a record.

Attributes:

  • model (FunctionModel) –

    The pydantic-ai FunctionModel to give a task model.

  • calls (list[AgentInfo]) –

    What the agent offered at each request: its output tools and settings.

  • messages (list[list[ModelMessage]]) –

    The messages of each request, oldest first.

ReplayCase dataclass

ReplayCase(expected: str, choice: str, probability: float | None, reordered_choice: str | None)

One case of a replay: the answer expected, and the one the model gave.

Attributes:

  • expected (str) –

    The option the case expects.

  • choice (str) –

    The option the model picked.

  • probability (float | None) –

    The probability the model gave its pick, or None on a language model.

  • reordered_choice (str | None) –

    The option it picked with the options in the reverse order, or None when the order was not checked.

  • is_correct (bool) –

    Whether the pick is the option expected.

is_correct property

is_correct: bool

Whether the pick is the option expected.

Coverage dataclass

Coverage(threshold: float, coverage: float, accuracy: float | None)

How a threshold of probability would work: how many cases it answers, how many correctly.

Attributes:

  • threshold (float) –

    The least probability a pick needs to be acted on.

  • coverage (float) –

    The share of the cases whose pick reaches the threshold.

  • accuracy (float | None) –

    The share of those that are correct, or None when none reach it.

ReplayReport dataclass

ReplayReport(cases: tuple[ReplayCase, ...])

What a replay found: every case, and the figures that choose a threshold.

Attributes:

  • cases (tuple[ReplayCase, ...]) –

    Every case, in the order of the file.

  • accuracy (float) –

    The share of the cases the model got right.

  • majority_baseline (float) –

    The accuracy of always picking the option most cases expect: the figure the model has to beat.

  • calibration_error (float | None) –

    The expected calibration error over ten bins of probability: how far the probabilities are from how often the picks are right, from 0.0 (the same) up. None on a language model, which gives no probabilities.

  • order_sensitivity (float | None) –

    The share of the cases whose pick changed when the options were given in the reverse order, or None when that was not checked.

Examples:

from xnano.ai.models.decisions import ChoiceModel
from xnano.ai.testing import ScriptedDecisionModel, replay

model = ScriptedDecisionModel({'response': {'left': 0.8, 'right': 0.2}})
cases = [
    {'evidence': {'wall': 'right'}, 'options': ['left', 'right'], 'expected': 'left'},
    {'evidence': {'wall': 'left'}, 'options': ['left', 'right'], 'expected': 'right'},
]
report = replay(ChoiceModel(model), cases, should_check_order=False)
assert report.accuracy == 0.5
assert report.at_threshold(0.7).coverage == 1.0

Methods:

  • at_threshold –

    How acting only on picks of at least threshold would have worked.

accuracy property

accuracy: float

The share of the cases the model got right.

majority_baseline property

majority_baseline: float

The accuracy of always picking the option most cases expect.

calibration_error property

calibration_error: float | None

The expected calibration error over ten bins of probability, or None.

order_sensitivity property

order_sensitivity: float | None

The share of the cases whose pick changed with the options reversed, or None.

at_threshold

at_threshold(threshold: float) -> Coverage

How acting only on picks of at least threshold would have worked.

Parameters:

  • threshold (float) –

    The least probability, from 0.0 to 1.0.

Raises:

Source code in xnano/ai/testing.py
def at_threshold(self, threshold: float) -> Coverage:
    """
    How acting only on picks of at least ``threshold`` would have worked.

    Args:
        threshold: The least probability, from ``0.0`` to ``1.0``.

    Raises:
        ValueError: ``threshold`` is outside ``0.0`` to ``1.0``.
    """
    if isinstance(threshold, bool) or not 0.0 <= threshold <= 1.0:
        raise ValueError(f'threshold must be from 0.0 to 1.0, but was {threshold!r}.')
    reached = [
        case
        for case in self.cases
        if case.probability is not None and case.probability >= threshold
    ]
    return Coverage(
        threshold=threshold,
        coverage=len(reached) / len(self.cases) if self.cases else 0.0,
        accuracy=_share(case.is_correct for case in reached) if reached else None,
    )

scripted_model

scripted_model(*steps: object, repeat_last: bool = False) -> ScriptedModel

A language model that answers each request with the next step of a script.

A step is text, a mapping of the arguments of the output (such as {'response': 'left'}), a (tool name, arguments) pair, a pydantic-ai ModelResponse, or a function of the messages and the AgentInfo that returns one of these.

Parameters:

  • steps (object, default: () ) –

    The answers, in order.

  • repeat_last (bool, default: False ) –

    Whether the last step answers every request after it.

Returns:

Examples:

from xnano.ai.models.text import TextModel
from xnano.ai.testing import scripted_model

script = scripted_model('The door creaks open.')
writer = TextModel(script.model)
assert writer.generate('Describe the door.') == 'The door creaks open.'
Source code in xnano/ai/testing.py
def scripted_model(*steps: object, repeat_last: bool = False) -> ScriptedModel:
    """
    A language model that answers each request with the next step of a script.

    A step is text, a mapping of the arguments of the output (such as ``{'response':
    'left'}``), a ``(tool name, arguments)`` pair, a pydantic-ai ``ModelResponse``, or a function
    of the messages and the ``AgentInfo`` that returns one of these.

    Args:
        steps: The answers, in order.
        repeat_last: Whether the last step answers every request after it.

    Returns:
        The [`ScriptedModel`][xnano.ai.testing.ScriptedModel]: give its ``model`` to a task
        model.

    Examples:
        ```python
        from xnano.ai.models.text import TextModel
        from xnano.ai.testing import scripted_model

        script = scripted_model('The door creaks open.')
        writer = TextModel(script.model)
        assert writer.generate('Describe the door.') == 'The door creaks open.'
        ```
    """
    queue = list(steps)
    record = ScriptedModel(
        model=FunctionModel(lambda messages, info: _next(messages, info))
    )

    def _next(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
        record.calls.append(info)
        record.messages.append(list(messages))
        if not queue:
            raise AssertionError('The scripted model has no step left for this request.')
        step = queue[0] if repeat_last and len(queue) == 1 else queue.pop(0)
        if callable(step) and not isinstance(step, (BaseModel, type)):
            step = step(messages, info)
        return _step_response(step, info)

    async def _stream(
        messages: list[ModelMessage], info: AgentInfo
    ) -> AsyncIterator[str | DeltaToolCalls]:
        response = _next(messages, info)
        for part in response.parts:
            if isinstance(part, TextPart):
                # Word by word, the way a model streams its text.
                for piece in re.findall(r'\S+\s*|\s+', part.content):
                    yield piece
            elif isinstance(part, ToolCallPart):
                yield {
                    0: DeltaToolCall(
                        name=part.tool_name, json_args=part.args_as_json_str()
                    )
                }

    record.model = FunctionModel(_next, stream_function=_stream, model_name='scripted')
    return record

replay_async

replay_async(model: ChoiceModel, cases: str | PathLike[str] | Sequence[Mapping[str, Any]], *, should_check_order: bool = True) -> Coroutine[Any, Any, ReplayReport]

Run cases through a choice model once, and report how it did, with await.

Each case is a mapping, or a line of a JSONL file, with evidence (what the model judges), options (a list of labels, or a mapping from label to what it means), the expected label and, when it differs from the model's, a question. The model is asked once per case, and once more with the options in the reverse order unless should_check_order is false; every figure of the report comes from those answers, with no new requests.

Parameters:

Returns:

Raises:

  • ValueError –

    A case has no evidence, options or expected, or expects a label that is not one of its options.

Source code in xnano/ai/testing.py
def replay_async(
    model: ChoiceModel,
    cases: str | os.PathLike[str] | Sequence[Mapping[str, Any]],
    *,
    should_check_order: bool = True,
) -> Coroutine[Any, Any, ReplayReport]:
    """
    Run cases through a choice model once, and report how it did, with ``await``.

    Each case is a mapping, or a line of a JSONL file, with ``evidence`` (what the model
    judges), ``options`` (a list of labels, or a mapping from label to what it means), the
    ``expected`` label and, when it differs from the model's, a ``question``. The model is
    asked once per case, and once more with the options in the reverse order unless
    ``should_check_order`` is false; every figure of the report comes from those answers, with
    no new requests.

    Args:
        model: The [`ChoiceModel`][xnano.ai.models.decisions.ChoiceModel] to replay.
        cases: The cases, or the path of a JSONL file of them.
        should_check_order: Whether to ask each case again with the options reversed.

    Returns:
        A coroutine that gives the [`ReplayReport`][xnano.ai.testing.ReplayReport].

    Raises:
        ValueError: A case has no ``evidence``, ``options`` or ``expected``, or expects a
            label that is not one of its options.
    """
    loaded = _load_cases(cases)
    return _replay(model, loaded, should_check_order)

replay

replay(model: ChoiceModel, cases: str | PathLike[str] | Sequence[Mapping[str, Any]], *, should_check_order: bool = True) -> ReplayReport

Run cases through a choice model once, and report how it did, waiting for it.

The same as replay_async.

Source code in xnano/ai/testing.py
def replay(
    model: ChoiceModel,
    cases: str | os.PathLike[str] | Sequence[Mapping[str, Any]],
    *,
    should_check_order: bool = True,
) -> ReplayReport:
    """
    Run cases through a choice model once, and report how it did, waiting for it.

    The same as [`replay_async`][xnano.ai.testing.replay_async].
    """
    from xnano.ai.models import _run_blocking

    return _run_blocking(
        replay_async(model, cases, should_check_order=should_check_order), 'replay'
    )