Skip to content

xnano.ai.capabilities

xnano.ai.capabilities

xnano.ai.capabilities

Capabilities for any pydantic-ai Agent: tools that let it see and change a canvas, draw pixel art and play melodies.

Give them to an agent and it is the agent of a terminal program: no session class is needed.

from pydantic_ai import Agent
from xnano.ai.capabilities import ArtCapability, CanvasCapability, MusicCapability

agent = Agent(
    'openrouter:anthropic/claude-sonnet-5-5',
    capabilities=[CanvasCapability(terminal, keyboard=True), ArtCapability(), MusicCapability()],
)

Every tool checks its arguments the way the task models check their answers: a fault goes back to the agent with what is wrong, so it can fix it in the same run. The tools run on the event loop of the agent's run: run the agent with run_sync or await agent.run(...) from the thread of a canvas from new(); a canvas on the screen can be used from any thread.

Examples:

import xnano
from pydantic_ai import Agent
from xnano.ai.capabilities import CanvasCapability
from xnano.ai.testing import scripted_model

script = scripted_model(
    ('write_component', {'program': "canvas.screen.write('hello')", 'width': 8, 'height': 1}),
    'Done.',
)
with xnano.new(width=20, height=3) as canvas:
    agent = Agent(script.model, capabilities=[CanvasCapability(canvas)])
    assert agent.run_sync('Say hello.').output == 'Done.'
    assert canvas.screen.render().splitlines()[0] == 'hello'

Classes:

  • CanvasCapability –

    Let an agent see a canvas and change it: write widgets and art, and, when allowed, press

  • ArtCapability –

    Let an agent draw and edit pixel art, kept by name in symbols.

  • MusicCapability –

    Let an agent play melodies on a canvas's speaker, and keep them by name.

CanvasCapability dataclass

CanvasCapability(canvas: Canvas[Any, Any], keyboard: bool = False, mouse: bool = False, components: dict[str, Placement[Any, Any]] = dict(), _repairs: list[str] = list())

Bases: AbstractCapability[Any]

Let an agent see a canvas and change it: write widgets and art, and, when allowed, press keys and use the mouse.

The agent's tools are observe (the canvas as text, under a column ruler with row numbers), write_component (a small program that draws a widget, written on the next free line unless it says where), update_component and show_art. With keyboard=True it also has press_key, type_text and paste; with mouse=True, click, drag and scroll at cell coordinates. Every action lets the canvas draw what changed and returns what the canvas then shows. The agent cannot press ctrl+c or a key that run(stop_on=...) ends the program with.

Parameters:

  • canvas (Canvas[Any, Any]) –

    The canvas the agent sees and changes.

  • keyboard (bool, default: False ) –

    Whether the agent can press keys, type and paste.

  • mouse (bool, default: False ) –

    Whether the agent can click, drag and scroll.

Attributes:

  • canvas (Canvas[Any, Any]) –

    The canvas the agent sees and changes.

  • keyboard (bool) –

    Whether the agent can press keys, type and paste.

  • mouse (bool) –

    Whether the agent can click, drag and scroll.

  • components (dict[str, Placement[Any, Any]]) –

    What the agent wrote, by the name it gave.

Methods:

  • get_serialization_name –

    A canvas cannot be written in an agent spec, so this capability has no name there.

  • get_instructions –

    Tell the agent what it can see and do.

  • get_toolset –

    The tools of the canvas.

  • observe –

    See the canvas: its text under a column ruler, with the row numbers.

  • write_component –

    Draw a widget: a Python program that writes on a canvas of width by height cells.

  • update_component –

    Draw a widget written with write_component again, with a new program, in its place.

  • show_art –

    Show pixel art: a palette of letters with roles, a size, and a program that paints them.

  • press_key –

    Press a key, such as 'enter', 'tab', 'a' or a binding such as 'ctrl+s'.

  • type_text –

    Type text, one key per character, as the person at the keyboard would.

  • paste –

    Paste text, which arrives whole, as a paste from the clipboard does.

  • click –

    Click a cell.

  • drag –

    Drag with the left button held, from one cell to another.

  • scroll –

    Turn the mouse wheel over a cell.

get_serialization_name classmethod

get_serialization_name() -> str | None

A canvas cannot be written in an agent spec, so this capability has no name there.

Source code in xnano/ai/capabilities.py
@classmethod
def get_serialization_name(cls) -> str | None:
    """A canvas cannot be written in an agent spec, so this capability has no name there."""
    return None

get_instructions

get_instructions() -> str

Tell the agent what it can see and do.

Source code in xnano/ai/capabilities.py
def get_instructions(self) -> str:
    """Tell the agent what it can see and do."""
    size = self.canvas.screen.size
    parts = [
        f'You can see and change a terminal canvas of {size.width} by {size.height} '
        f'cells. Call observe to see it. write_component runs a small Python program '
        f'that draws a widget with canvas.screen.write(text, x=..., y=..., border=...); '
        f'show_art shows pixel art.'
    ]
    if self.keyboard:
        parts.append(
            'press_key, type_text and paste act as the person at the keyboard.'
        )
    if self.mouse:
        parts.append('click, drag and scroll act as the mouse, at cell coordinates.')
    return ' '.join(parts)

get_toolset

get_toolset() -> FunctionToolset[Any]

The tools of the canvas.

Source code in xnano/ai/capabilities.py
def get_toolset(self) -> FunctionToolset[Any]:
    """The tools of the canvas."""
    tools: list[Any] = [
        self.observe,
        self.write_component,
        self.update_component,
        self.show_art,
    ]
    if self.keyboard:
        tools.extend((self.press_key, self.type_text, self.paste))
    if self.mouse:
        tools.extend((self.click, self.drag, self.scroll))
    return FunctionToolset(tools)

observe async

observe() -> str

See the canvas: its text under a column ruler, with the row numbers.

Source code in xnano/ai/capabilities.py
async def observe(self) -> str:
    """See the canvas: its text under a column ruler, with the row numbers."""
    screen = self.canvas.screen
    return _ruled(screen.render(), screen.width)

write_component async

write_component(program: str, width: int, height: int, name: str | None = None, x: int | None = None, y: int | None = None) -> str

Draw a widget: a Python program that writes on a canvas of width by height cells.

Parameters:

  • program (str) –

    The program; it draws with canvas.screen.write(...).

  • width (int) –

    The width of the widget, in cells.

  • height (int) –

    The height of the widget, in rows.

  • name (str | None, default: None ) –

    A name to change it by later with update_component.

  • x (int | None, default: None ) –

    The column of its left edge; on the next free line when x and y are left out.

  • y (int | None, default: None ) –

    The row of its top edge.

Source code in xnano/ai/capabilities.py
async def write_component(
    self,
    program: str,
    width: int,
    height: int,
    name: str | None = None,
    x: int | None = None,
    y: int | None = None,
) -> str:
    """
    Draw a widget: a Python program that writes on a canvas of width by height cells.

    Args:
        program: The program; it draws with canvas.screen.write(...).
        width: The width of the widget, in cells.
        height: The height of the widget, in rows.
        name: A name to change it by later with update_component.
        x: The column of its left edge; on the next free line when x and y are left out.
        y: The row of its top edge.
    """
    component = self._component(program, width, height)
    placement = self.canvas.screen.write(
        component, x='next' if x is None else x, y='next' if y is None else y
    )
    if name is not None:
        previous = self.components.pop(name, None)
        if previous is not None:
            previous.remove()
        self.components[name] = placement
    return await self._settled()

update_component async

update_component(name: str, program: str) -> str

Draw a widget written with write_component again, with a new program, in its place.

Parameters:

  • name (str) –

    The name it was written with.

  • program (str) –

    The new program.

Source code in xnano/ai/capabilities.py
async def update_component(self, name: str, program: str) -> str:
    """
    Draw a widget written with write_component again, with a new program, in its place.

    Args:
        name: The name it was written with.
        program: The new program.
    """
    placement = self.components.get(name)
    if placement is None or placement._removed:
        known = ', '.join(sorted(self.components)) or 'none yet'
        raise ModelRetry(
            f'There is no widget called {name!r}. The widgets are: {known}. Write it with '
            f'write_component first.'
        )
    component = self._component(program, placement.width, placement.height)
    x, y = placement.x, placement.y
    placement.remove()
    self.components[name] = self.canvas.screen.write(component, x=x, y=y)
    return await self._settled()

show_art async

show_art(name: str, palette: list[_Color], width: int, height: int, program: str, x: int | None = None, y: int | None = None) -> str

Show pixel art: a palette of letters with roles, a size, and a program that paints them.

The program paints W by H transparent pixels with pixel(x, y, letter), rect(x, y, width, height, letter), ellipse(x, y, width, height, letter), line(x1, y1, x2, y2, letter), polygon(points, letter), fill(x, y, letter), mirror_x(), mirror_y() and outline(letter). A pixel is half a cell tall. The art is kept under the name, so the program finds it with xnano.symbols.lookup(name).

Parameters:

  • name (str) –

    The name to keep the art under.

  • palette (list[_Color]) –

    Each letter, what its colour is for, and the colour as #rrggbb.

  • width (int) –

    The width in pixels.

  • height (int) –

    The height in pixels.

  • program (str) –

    The program that paints the palette's letters.

  • x (int | None, default: None ) –

    The column of its left edge; on the next free line when x and y are left out.

  • y (int | None, default: None ) –

    The row of its top edge.

Source code in xnano/ai/capabilities.py
async def show_art(
    self,
    name: str,
    palette: list[_Color],
    width: int,
    height: int,
    program: str,
    x: int | None = None,
    y: int | None = None,
) -> str:
    """
    Show pixel art: a palette of letters with roles, a size, and a program that paints them.

    The program paints W by H transparent pixels with pixel(x, y, letter), rect(x, y,
    width, height, letter), ellipse(x, y, width, height, letter), line(x1, y1, x2, y2,
    letter), polygon(points, letter), fill(x, y, letter), mirror_x(), mirror_y() and
    outline(letter). A pixel is half a cell tall. The art is kept under the name, so the
    program finds it with xnano.symbols.lookup(name).

    Args:
        name: The name to keep the art under.
        palette: Each letter, what its colour is for, and the colour as #rrggbb.
        width: The width in pixels.
        height: The height in pixels.
        program: The program that paints the palette's letters.
        x: The column of its left edge; on the next free line when x and y are left out.
        y: The row of its top edge.
    """
    art = _checked_art(palette, width, height, program, name=name)
    self.canvas.screen.write(
        art, x='next' if x is None else x, y='next' if y is None else y
    )
    return await self._settled()

press_key async

press_key(key: str) -> str

Press a key, such as 'enter', 'tab', 'a' or a binding such as 'ctrl+s'.

Parameters:

  • key (str) –

    The key, or the key binding.

Source code in xnano/ai/capabilities.py
async def press_key(self, key: str) -> str:
    """
    Press a key, such as 'enter', 'tab', 'a' or a binding such as 'ctrl+s'.

    Args:
        key: The key, or the key binding.
    """
    normalized = key.strip().lower()
    if normalized == 'ctrl+c' or normalized in self.canvas._stop_keys:
        raise ModelRetry(
            f'{key!r} ends the program, so it cannot be pressed from here. Press another '
            f'key.'
        )
    try:
        self.canvas.keyboard.press(key)
    except (TypeError, ValueError) as error:
        raise _refusal(error) from None
    return await self._settled()

type_text async

type_text(text: str) -> str

Type text, one key per character, as the person at the keyboard would.

Parameters:

  • text (str) –

    The text to type.

Source code in xnano/ai/capabilities.py
async def type_text(self, text: str) -> str:
    """
    Type text, one key per character, as the person at the keyboard would.

    Args:
        text: The text to type.
    """
    for character in text:
        name = {' ': 'space', '\n': 'enter', '\t': 'tab'}.get(character, character)
        try:
            self.canvas.keyboard.press(name)
        except (TypeError, ValueError) as error:
            raise _refusal(error) from None
    return await self._settled()

paste async

paste(text: str) -> str

Paste text, which arrives whole, as a paste from the clipboard does.

Parameters:

  • text (str) –

    The text to paste.

Source code in xnano/ai/capabilities.py
async def paste(self, text: str) -> str:
    """
    Paste text, which arrives whole, as a paste from the clipboard does.

    Args:
        text: The text to paste.
    """
    self.canvas.keyboard.paste(text)
    return await self._settled()

click async

click(x: int, y: int, button: Literal['left', 'right', 'middle'] = 'left') -> str

Click a cell.

Parameters:

  • x (int) –

    The column, from 0 at the left.

  • y (int) –

    The row, from 0 at the top.

  • button (Literal['left', 'right', 'middle'], default: 'left' ) –

    The mouse button.

Source code in xnano/ai/capabilities.py
async def click(
    self, x: int, y: int, button: Literal['left', 'right', 'middle'] = 'left'
) -> str:
    """
    Click a cell.

    Args:
        x: The column, from 0 at the left.
        y: The row, from 0 at the top.
        button: The mouse button.
    """
    self._check_cell(x, y)
    self.canvas.mouse.click(button, x=x, y=y)
    return await self._settled()

drag async

drag(from_x: int, from_y: int, to_x: int, to_y: int) -> str

Drag with the left button held, from one cell to another.

Parameters:

  • from_x (int) –

    The column to start at.

  • from_y (int) –

    The row to start at.

  • to_x (int) –

    The column to end at.

  • to_y (int) –

    The row to end at.

Source code in xnano/ai/capabilities.py
async def drag(self, from_x: int, from_y: int, to_x: int, to_y: int) -> str:
    """
    Drag with the left button held, from one cell to another.

    Args:
        from_x: The column to start at.
        from_y: The row to start at.
        to_x: The column to end at.
        to_y: The row to end at.
    """
    self._check_cell(from_x, from_y)
    self._check_cell(to_x, to_y)
    with self.canvas.mouse.press('left', x=from_x, y=from_y) as held:
        held.move(to_x - from_x, to_y - from_y)
    return await self._settled()

scroll async

scroll(x: int, y: int, rows: int) -> str

Turn the mouse wheel over a cell.

Parameters:

  • x (int) –

    The column of the pointer.

  • y (int) –

    The row of the pointer.

  • rows (int) –

    How far: a positive number scrolls down, a negative one up.

Source code in xnano/ai/capabilities.py
async def scroll(self, x: int, y: int, rows: int) -> str:
    """
    Turn the mouse wheel over a cell.

    Args:
        x: The column of the pointer.
        y: The row of the pointer.
        rows: How far: a positive number scrolls down, a negative one up.
    """
    self._check_cell(x, y)
    mouse = self.canvas.mouse
    mouse.pointer = (x, y)
    mouse.scroll.turn(y=rows)
    return await self._settled()

ArtCapability dataclass

ArtCapability(max_colors: int = 16)

Bases: AbstractCapability[Any]

Let an agent draw and edit pixel art, kept by name in symbols.

The agent's tools are draw_art (a palette of letters with roles, a size, and a short program of drawing calls that paints the letters) and edit_art (runs of letters that change pixels of a piece it drew). The program finds each piece with xnano.symbols.lookup(name, kind=xnano.art.Art) and writes it where it wants.

Parameters:

  • max_colors (int, default: 16 ) –

    The most colours a piece may have.

Attributes:

  • max_colors (int) –

    The most colours a piece may have.

Methods:

get_serialization_name classmethod

get_serialization_name() -> str | None

This capability has no name in an agent spec.

Source code in xnano/ai/capabilities.py
@classmethod
def get_serialization_name(cls) -> str | None:
    """This capability has no name in an agent spec."""
    return None

get_instructions

get_instructions() -> str

Tell the agent how to draw.

Source code in xnano/ai/capabilities.py
def get_instructions(self) -> str:
    """Tell the agent how to draw."""
    return (
        'draw_art draws pixel art: name the colours by role (outline, base, shadow, '
        'highlight, accent, skin, detail) under single lowercase letters, then write a '
        'short program that paints them on W by H transparent pixels with pixel(x, y, '
        'letter), rect(x, y, width, height, letter), ellipse(x, y, width, height, '
        'letter), line(x1, y1, x2, y2, letter), polygon(points, letter), fill(x, y, '
        'letter), mirror_x(), mirror_y() and outline(letter). Light comes from the top '
        f'left; use one outline colour and at most {self.max_colors} colours. edit_art '
        'changes pixels of a piece already drawn.'
    )

get_toolset

get_toolset() -> FunctionToolset[Any]

The tools of pixel art.

Source code in xnano/ai/capabilities.py
def get_toolset(self) -> FunctionToolset[Any]:
    """The tools of pixel art."""
    return FunctionToolset([self.draw_art, self.edit_art])

draw_art async

draw_art(name: str, palette: list[_Color], width: int, height: int, program: str) -> str

Draw pixel art and keep it under a name.

Parameters:

  • name (str) –

    The name to keep the art under.

  • palette (list[_Color]) –

    Each letter, what its colour is for, and the colour as #rrggbb.

  • width (int) –

    The width in pixels.

  • height (int) –

    The height in pixels.

  • program (str) –

    The program that paints the palette's letters onto W by H pixels.

Source code in xnano/ai/capabilities.py
async def draw_art(
    self, name: str, palette: list[_Color], width: int, height: int, program: str
) -> str:
    """
    Draw pixel art and keep it under a name.

    Args:
        name: The name to keep the art under.
        palette: Each letter, what its colour is for, and the colour as #rrggbb.
        width: The width in pixels.
        height: The height in pixels.
        program: The program that paints the palette's letters onto W by H pixels.
    """
    art = _checked_art(
        palette, width, height, program, name=name, max_colors=self.max_colors
    )
    return f'Drew {name!r}, {art.width} by {art.height} pixels.'

edit_art async

edit_art(name: str, runs: list[_Run], new_colors: list[_Color]) -> str

Change pixels of a piece drawn with draw_art: runs of letters along rows.

Parameters:

  • name (str) –

    The name of the piece.

  • runs (list[_Run]) –

    Each run's row, its first column, and its letters.

  • new_colors (list[_Color]) –

    Colours the runs use that the palette does not have yet.

Source code in xnano/ai/capabilities.py
async def edit_art(
    self, name: str, runs: list[_Run], new_colors: list[_Color]
) -> str:
    """
    Change pixels of a piece drawn with draw_art: runs of letters along rows.

    Args:
        name: The name of the piece.
        runs: Each run's row, its first column, and its letters.
        new_colors: Colours the runs use that the palette does not have yet.
    """
    try:
        art = symbols.lookup(name, kind=Art)
    except (KeyError, LookupError, TypeError, ValueError) as error:
        raise _refusal(error) from None
    text = ArtText.from_art(art)
    checker = _EditChecker(_Attempts(), text, frozenset(), True, 1.0, 'patch')
    edited = checker.patch(runs, new_colors)
    edited.to_art(symbol=name)
    return f'Changed {name!r}.\n{edited.labelled()}'

MusicCapability dataclass

MusicCapability(canvas: Canvas[Any, Any] | None = None)

Bases: AbstractCapability[Any]

Let an agent play melodies on a canvas's speaker, and keep them by name.

The agent's tools are play_melody and compose_melody, both in xnano's melody text ('C4 E4 G4:2 | rest C5+E5+G5'). A melody kept by name plays with play(name). Needs the audio extra.

Parameters:

  • canvas (Canvas[Any, Any] | None, default: None ) –

    The canvas whose speaker plays; None plays with play.

Attributes:

  • canvas (Canvas[Any, Any] | None) –

    The canvas whose speaker plays.

Methods:

get_serialization_name classmethod

get_serialization_name() -> str | None

This capability has no name in an agent spec.

Source code in xnano/ai/capabilities.py
@classmethod
def get_serialization_name(cls) -> str | None:
    """This capability has no name in an agent spec."""
    return None

get_instructions

get_instructions() -> str

Tell the agent how to write melodies.

Source code in xnano/ai/capabilities.py
def get_instructions(self) -> str:
    """Tell the agent how to write melodies."""
    from xnano.ai.models.music import _GRAMMAR

    return f'play_melody plays a melody and compose_melody keeps one by name.\n{_GRAMMAR}'

get_toolset

get_toolset() -> FunctionToolset[Any]

The tools of music.

Source code in xnano/ai/capabilities.py
def get_toolset(self) -> FunctionToolset[Any]:
    """The tools of music."""
    return FunctionToolset([self.play_melody, self.compose_melody])

play_melody async

play_melody(text: str, bpm: float = 120.0) -> str

Play a melody now.

Parameters:

  • text (str) –

    The melody text.

  • bpm (float, default: 120.0 ) –

    The tempo, in beats per minute.

Source code in xnano/ai/capabilities.py
async def play_melody(self, text: str, bpm: float = 120.0) -> str:
    """
    Play a melody now.

    Args:
        text: The melody text.
        bpm: The tempo, in beats per minute.
    """
    melody = _checked_melody(text, bpm, symbol=None)
    if self.canvas is None:
        from xnano.audio import play

        play(melody)
    else:
        self.canvas.speaker.play(melody)
    return f'Playing {melody.length_beats:g} beats.'

compose_melody async

compose_melody(name: str, text: str, bpm: float = 120.0) -> str

Keep a melody under a name, so the program plays it with play(name).

Parameters:

  • name (str) –

    The name to keep the melody under.

  • text (str) –

    The melody text.

  • bpm (float, default: 120.0 ) –

    The tempo, in beats per minute.

Source code in xnano/ai/capabilities.py
async def compose_melody(self, name: str, text: str, bpm: float = 120.0) -> str:
    """
    Keep a melody under a name, so the program plays it with play(name).

    Args:
        name: The name to keep the melody under.
        text: The melody text.
        bpm: The tempo, in beats per minute.
    """
    melody = _checked_melody(text, bpm, symbol=name)
    return f'Kept {name!r}, {melody.length_beats:g} beats.'