xnano.ai.capabilities
xnano.ai.capabilities
¶
xnano.ai.capabilities
Capabilities for any pydantic-ai Agent: tools that let it see and change a canvas, draw
pixel art and play melodies.
Give them to an agent and it is the agent of a terminal program: no session class is needed.
from pydantic_ai import Agent
from xnano.ai.capabilities import ArtCapability, CanvasCapability, MusicCapability
agent = Agent(
'openrouter:anthropic/claude-sonnet-5-5',
capabilities=[CanvasCapability(terminal, keyboard=True), ArtCapability(), MusicCapability()],
)
Every tool checks its arguments the way the task models check their answers: a fault goes back
to the agent with what is wrong, so it can fix it in the same run. The tools run on the event
loop of the agent's run: run the agent with run_sync or await agent.run(...) from the
thread of a canvas from new(); a canvas on the screen can be used from any thread.
Examples:
import xnano
from pydantic_ai import Agent
from xnano.ai.capabilities import CanvasCapability
from xnano.ai.testing import scripted_model
script = scripted_model(
('write_component', {'program': "canvas.screen.write('hello')", 'width': 8, 'height': 1}),
'Done.',
)
with xnano.new(width=20, height=3) as canvas:
agent = Agent(script.model, capabilities=[CanvasCapability(canvas)])
assert agent.run_sync('Say hello.').output == 'Done.'
assert canvas.screen.render().splitlines()[0] == 'hello'
Classes:
-
CanvasCapability–Let an agent see a canvas and change it: write widgets and art, and, when allowed, press
-
ArtCapability–Let an agent draw and edit pixel art, kept by name in
symbols. -
MusicCapability–Let an agent play melodies on a canvas's speaker, and keep them by name.
CanvasCapability
dataclass
¶
CanvasCapability(canvas: Canvas[Any, Any], keyboard: bool = False, mouse: bool = False, components: dict[str, Placement[Any, Any]] = dict(), _repairs: list[str] = list())
Bases: AbstractCapability[Any]
Let an agent see a canvas and change it: write widgets and art, and, when allowed, press keys and use the mouse.
The agent's tools are observe (the canvas as text, under a column ruler with row
numbers), write_component (a small program that draws a widget, written on the next
free line unless it says where), update_component and show_art. With
keyboard=True it also has press_key, type_text and paste; with
mouse=True, click, drag and scroll at cell coordinates. Every action lets
the canvas draw what changed and returns what the canvas then shows. The agent cannot
press ctrl+c or a key that run(stop_on=...) ends the program with.
Parameters:
-
canvas(Canvas[Any, Any]) –The canvas the agent sees and changes.
-
keyboard(bool, default:False) –Whether the agent can press keys, type and paste.
-
mouse(bool, default:False) –Whether the agent can click, drag and scroll.
Attributes:
-
canvas(Canvas[Any, Any]) –The canvas the agent sees and changes.
-
keyboard(bool) –Whether the agent can press keys, type and paste.
-
mouse(bool) –Whether the agent can click, drag and scroll.
-
components(dict[str, Placement[Any, Any]]) –What the agent wrote, by the name it gave.
Methods:
-
get_serialization_name–A canvas cannot be written in an agent spec, so this capability has no name there.
-
get_instructions–Tell the agent what it can see and do.
-
get_toolset–The tools of the canvas.
-
observe–See the canvas: its text under a column ruler, with the row numbers.
-
write_component–Draw a widget: a Python program that writes on a canvas of width by height cells.
-
update_component–Draw a widget written with write_component again, with a new program, in its place.
-
show_art–Show pixel art: a palette of letters with roles, a size, and a program that paints them.
-
press_key–Press a key, such as 'enter', 'tab', 'a' or a binding such as 'ctrl+s'.
-
type_text–Type text, one key per character, as the person at the keyboard would.
-
paste–Paste text, which arrives whole, as a paste from the clipboard does.
-
click–Click a cell.
-
drag–Drag with the left button held, from one cell to another.
-
scroll–Turn the mouse wheel over a cell.
get_instructions
¶
get_instructions() -> str
Tell the agent what it can see and do.
Source code in xnano/ai/capabilities.py
def get_instructions(self) -> str:
"""Tell the agent what it can see and do."""
size = self.canvas.screen.size
parts = [
f'You can see and change a terminal canvas of {size.width} by {size.height} '
f'cells. Call observe to see it. write_component runs a small Python program '
f'that draws a widget with canvas.screen.write(text, x=..., y=..., border=...); '
f'show_art shows pixel art.'
]
if self.keyboard:
parts.append(
'press_key, type_text and paste act as the person at the keyboard.'
)
if self.mouse:
parts.append('click, drag and scroll act as the mouse, at cell coordinates.')
return ' '.join(parts)
get_toolset
¶
get_toolset() -> FunctionToolset[Any]
The tools of the canvas.
Source code in xnano/ai/capabilities.py
def get_toolset(self) -> FunctionToolset[Any]:
"""The tools of the canvas."""
tools: list[Any] = [
self.observe,
self.write_component,
self.update_component,
self.show_art,
]
if self.keyboard:
tools.extend((self.press_key, self.type_text, self.paste))
if self.mouse:
tools.extend((self.click, self.drag, self.scroll))
return FunctionToolset(tools)
write_component
async
¶
write_component(program: str, width: int, height: int, name: str | None = None, x: int | None = None, y: int | None = None) -> str
Draw a widget: a Python program that writes on a canvas of width by height cells.
Parameters:
-
program(str) –The program; it draws with canvas.screen.write(...).
-
width(int) –The width of the widget, in cells.
-
height(int) –The height of the widget, in rows.
-
name(str | None, default:None) –A name to change it by later with update_component.
-
x(int | None, default:None) –The column of its left edge; on the next free line when x and y are left out.
-
y(int | None, default:None) –The row of its top edge.
Source code in xnano/ai/capabilities.py
async def write_component(
self,
program: str,
width: int,
height: int,
name: str | None = None,
x: int | None = None,
y: int | None = None,
) -> str:
"""
Draw a widget: a Python program that writes on a canvas of width by height cells.
Args:
program: The program; it draws with canvas.screen.write(...).
width: The width of the widget, in cells.
height: The height of the widget, in rows.
name: A name to change it by later with update_component.
x: The column of its left edge; on the next free line when x and y are left out.
y: The row of its top edge.
"""
component = self._component(program, width, height)
placement = self.canvas.screen.write(
component, x='next' if x is None else x, y='next' if y is None else y
)
if name is not None:
previous = self.components.pop(name, None)
if previous is not None:
previous.remove()
self.components[name] = placement
return await self._settled()
update_component
async
¶
Draw a widget written with write_component again, with a new program, in its place.
Parameters:
Source code in xnano/ai/capabilities.py
async def update_component(self, name: str, program: str) -> str:
"""
Draw a widget written with write_component again, with a new program, in its place.
Args:
name: The name it was written with.
program: The new program.
"""
placement = self.components.get(name)
if placement is None or placement._removed:
known = ', '.join(sorted(self.components)) or 'none yet'
raise ModelRetry(
f'There is no widget called {name!r}. The widgets are: {known}. Write it with '
f'write_component first.'
)
component = self._component(program, placement.width, placement.height)
x, y = placement.x, placement.y
placement.remove()
self.components[name] = self.canvas.screen.write(component, x=x, y=y)
return await self._settled()
show_art
async
¶
show_art(name: str, palette: list[_Color], width: int, height: int, program: str, x: int | None = None, y: int | None = None) -> str
Show pixel art: a palette of letters with roles, a size, and a program that paints them.
The program paints W by H transparent pixels with pixel(x, y, letter), rect(x, y, width, height, letter), ellipse(x, y, width, height, letter), line(x1, y1, x2, y2, letter), polygon(points, letter), fill(x, y, letter), mirror_x(), mirror_y() and outline(letter). A pixel is half a cell tall. The art is kept under the name, so the program finds it with xnano.symbols.lookup(name).
Parameters:
-
name(str) –The name to keep the art under.
-
palette(list[_Color]) –Each letter, what its colour is for, and the colour as #rrggbb.
-
width(int) –The width in pixels.
-
height(int) –The height in pixels.
-
program(str) –The program that paints the palette's letters.
-
x(int | None, default:None) –The column of its left edge; on the next free line when x and y are left out.
-
y(int | None, default:None) –The row of its top edge.
Source code in xnano/ai/capabilities.py
async def show_art(
self,
name: str,
palette: list[_Color],
width: int,
height: int,
program: str,
x: int | None = None,
y: int | None = None,
) -> str:
"""
Show pixel art: a palette of letters with roles, a size, and a program that paints them.
The program paints W by H transparent pixels with pixel(x, y, letter), rect(x, y,
width, height, letter), ellipse(x, y, width, height, letter), line(x1, y1, x2, y2,
letter), polygon(points, letter), fill(x, y, letter), mirror_x(), mirror_y() and
outline(letter). A pixel is half a cell tall. The art is kept under the name, so the
program finds it with xnano.symbols.lookup(name).
Args:
name: The name to keep the art under.
palette: Each letter, what its colour is for, and the colour as #rrggbb.
width: The width in pixels.
height: The height in pixels.
program: The program that paints the palette's letters.
x: The column of its left edge; on the next free line when x and y are left out.
y: The row of its top edge.
"""
art = _checked_art(palette, width, height, program, name=name)
self.canvas.screen.write(
art, x='next' if x is None else x, y='next' if y is None else y
)
return await self._settled()
press_key
async
¶
Press a key, such as 'enter', 'tab', 'a' or a binding such as 'ctrl+s'.
Parameters:
-
key(str) –The key, or the key binding.
Source code in xnano/ai/capabilities.py
async def press_key(self, key: str) -> str:
"""
Press a key, such as 'enter', 'tab', 'a' or a binding such as 'ctrl+s'.
Args:
key: The key, or the key binding.
"""
normalized = key.strip().lower()
if normalized == 'ctrl+c' or normalized in self.canvas._stop_keys:
raise ModelRetry(
f'{key!r} ends the program, so it cannot be pressed from here. Press another '
f'key.'
)
try:
self.canvas.keyboard.press(key)
except (TypeError, ValueError) as error:
raise _refusal(error) from None
return await self._settled()
type_text
async
¶
Type text, one key per character, as the person at the keyboard would.
Parameters:
-
text(str) –The text to type.
Source code in xnano/ai/capabilities.py
async def type_text(self, text: str) -> str:
"""
Type text, one key per character, as the person at the keyboard would.
Args:
text: The text to type.
"""
for character in text:
name = {' ': 'space', '\n': 'enter', '\t': 'tab'}.get(character, character)
try:
self.canvas.keyboard.press(name)
except (TypeError, ValueError) as error:
raise _refusal(error) from None
return await self._settled()
paste
async
¶
Paste text, which arrives whole, as a paste from the clipboard does.
Parameters:
-
text(str) –The text to paste.
click
async
¶
Click a cell.
Parameters:
-
x(int) –The column, from 0 at the left.
-
y(int) –The row, from 0 at the top.
-
button(Literal['left', 'right', 'middle'], default:'left') –The mouse button.
Source code in xnano/ai/capabilities.py
async def click(
self, x: int, y: int, button: Literal['left', 'right', 'middle'] = 'left'
) -> str:
"""
Click a cell.
Args:
x: The column, from 0 at the left.
y: The row, from 0 at the top.
button: The mouse button.
"""
self._check_cell(x, y)
self.canvas.mouse.click(button, x=x, y=y)
return await self._settled()
drag
async
¶
Drag with the left button held, from one cell to another.
Parameters:
-
from_x(int) –The column to start at.
-
from_y(int) –The row to start at.
-
to_x(int) –The column to end at.
-
to_y(int) –The row to end at.
Source code in xnano/ai/capabilities.py
async def drag(self, from_x: int, from_y: int, to_x: int, to_y: int) -> str:
"""
Drag with the left button held, from one cell to another.
Args:
from_x: The column to start at.
from_y: The row to start at.
to_x: The column to end at.
to_y: The row to end at.
"""
self._check_cell(from_x, from_y)
self._check_cell(to_x, to_y)
with self.canvas.mouse.press('left', x=from_x, y=from_y) as held:
held.move(to_x - from_x, to_y - from_y)
return await self._settled()
scroll
async
¶
Turn the mouse wheel over a cell.
Parameters:
-
x(int) –The column of the pointer.
-
y(int) –The row of the pointer.
-
rows(int) –How far: a positive number scrolls down, a negative one up.
Source code in xnano/ai/capabilities.py
async def scroll(self, x: int, y: int, rows: int) -> str:
"""
Turn the mouse wheel over a cell.
Args:
x: The column of the pointer.
y: The row of the pointer.
rows: How far: a positive number scrolls down, a negative one up.
"""
self._check_cell(x, y)
mouse = self.canvas.mouse
mouse.pointer = (x, y)
mouse.scroll.turn(y=rows)
return await self._settled()
ArtCapability
dataclass
¶
ArtCapability(max_colors: int = 16)
Bases: AbstractCapability[Any]
Let an agent draw and edit pixel art, kept by name in symbols.
The agent's tools are draw_art (a palette of letters with roles, a size, and a short
program of drawing calls that paints the letters) and edit_art (runs of letters that
change pixels of a piece it drew).
The program finds each piece with xnano.symbols.lookup(name, kind=xnano.art.Art) and
writes it where it wants.
Parameters:
-
max_colors(int, default:16) –The most colours a piece may have.
Attributes:
-
max_colors(int) –The most colours a piece may have.
Methods:
-
get_serialization_name–This capability has no name in an agent spec.
-
get_instructions–Tell the agent how to draw.
-
get_toolset–The tools of pixel art.
-
draw_art–Draw pixel art and keep it under a name.
-
edit_art–Change pixels of a piece drawn with draw_art: runs of letters along rows.
get_instructions
¶
get_instructions() -> str
Tell the agent how to draw.
Source code in xnano/ai/capabilities.py
def get_instructions(self) -> str:
"""Tell the agent how to draw."""
return (
'draw_art draws pixel art: name the colours by role (outline, base, shadow, '
'highlight, accent, skin, detail) under single lowercase letters, then write a '
'short program that paints them on W by H transparent pixels with pixel(x, y, '
'letter), rect(x, y, width, height, letter), ellipse(x, y, width, height, '
'letter), line(x1, y1, x2, y2, letter), polygon(points, letter), fill(x, y, '
'letter), mirror_x(), mirror_y() and outline(letter). Light comes from the top '
f'left; use one outline colour and at most {self.max_colors} colours. edit_art '
'changes pixels of a piece already drawn.'
)
draw_art
async
¶
Draw pixel art and keep it under a name.
Parameters:
-
name(str) –The name to keep the art under.
-
palette(list[_Color]) –Each letter, what its colour is for, and the colour as #rrggbb.
-
width(int) –The width in pixels.
-
height(int) –The height in pixels.
-
program(str) –The program that paints the palette's letters onto W by H pixels.
Source code in xnano/ai/capabilities.py
async def draw_art(
self, name: str, palette: list[_Color], width: int, height: int, program: str
) -> str:
"""
Draw pixel art and keep it under a name.
Args:
name: The name to keep the art under.
palette: Each letter, what its colour is for, and the colour as #rrggbb.
width: The width in pixels.
height: The height in pixels.
program: The program that paints the palette's letters onto W by H pixels.
"""
art = _checked_art(
palette, width, height, program, name=name, max_colors=self.max_colors
)
return f'Drew {name!r}, {art.width} by {art.height} pixels.'
edit_art
async
¶
Change pixels of a piece drawn with draw_art: runs of letters along rows.
Parameters:
-
name(str) –The name of the piece.
-
runs(list[_Run]) –Each run's row, its first column, and its letters.
-
new_colors(list[_Color]) –Colours the runs use that the palette does not have yet.
Source code in xnano/ai/capabilities.py
async def edit_art(
self, name: str, runs: list[_Run], new_colors: list[_Color]
) -> str:
"""
Change pixels of a piece drawn with draw_art: runs of letters along rows.
Args:
name: The name of the piece.
runs: Each run's row, its first column, and its letters.
new_colors: Colours the runs use that the palette does not have yet.
"""
try:
art = symbols.lookup(name, kind=Art)
except (KeyError, LookupError, TypeError, ValueError) as error:
raise _refusal(error) from None
text = ArtText.from_art(art)
checker = _EditChecker(_Attempts(), text, frozenset(), True, 1.0, 'patch')
edited = checker.patch(runs, new_colors)
edited.to_art(symbol=name)
return f'Changed {name!r}.\n{edited.labelled()}'
MusicCapability
dataclass
¶
Bases: AbstractCapability[Any]
Let an agent play melodies on a canvas's speaker, and keep them by name.
The agent's tools are play_melody and compose_melody, both in xnano's melody text
('C4 E4 G4:2 | rest C5+E5+G5'). A melody kept by name plays with play(name). Needs
the audio extra.
Parameters:
-
canvas(Canvas[Any, Any] | None, default:None) –The canvas whose speaker plays;
Noneplays withplay.
Attributes:
Methods:
-
get_serialization_name–This capability has no name in an agent spec.
-
get_instructions–Tell the agent how to write melodies.
-
get_toolset–The tools of music.
-
play_melody–Play a melody now.
-
compose_melody–Keep a melody under a name, so the program plays it with play(name).
play_melody
async
¶
Play a melody now.
Parameters:
Source code in xnano/ai/capabilities.py
async def play_melody(self, text: str, bpm: float = 120.0) -> str:
"""
Play a melody now.
Args:
text: The melody text.
bpm: The tempo, in beats per minute.
"""
melody = _checked_melody(text, bpm, symbol=None)
if self.canvas is None:
from xnano.audio import play
play(melody)
else:
self.canvas.speaker.play(melody)
return f'Playing {melody.length_beats:g} beats.'
compose_melody
async
¶
Keep a melody under a name, so the program plays it with play(name).
Parameters:
-
name(str) –The name to keep the melody under.
-
text(str) –The melody text.
-
bpm(float, default:120.0) –The tempo, in beats per minute.
Source code in xnano/ai/capabilities.py
async def compose_melody(self, name: str, text: str, bpm: float = 120.0) -> str:
"""
Keep a melody under a name, so the program plays it with play(name).
Args:
name: The name to keep the melody under.
text: The melody text.
bpm: The tempo, in beats per minute.
"""
melody = _checked_melody(text, bpm, symbol=name)
return f'Kept {name!r}, {melody.length_beats:g} beats.'