Does it do what you said? Each scenario is something people actually say to a whiteboard, paired with the graph that should exist afterwards. We score the resulting typed graph, never the agent’s words. Offline runs the deterministic command parser (it also runs in CI). Live sends the same turns to the AssemblyAI Voice Agent and applies its own tool calls.
Offline parser Live voice agent
—
Running
Postgres behind the API, queue between them “Add a Postgres behind the API, put a queue between them.” add, insert Add a single component “Add a Redis cache” add Add several components in one breath “Add a web app, an API gateway and an orders service” add, inference Quantities “Add a queue and three workers” add Infers the only service as the caller “We need a MongoDB” inference Connect two components “Add a payments service” → “Connect api to payments” connect Rejects a client talking to a database “Connect the web app straight to postgres” rules, rejection Rejects a database calling a service “Connect postgres to api” rules, rejection Links to a queue are async “Add Kafka” → “Connect api to kafka” rules, connect Cache in front of the database “Put a Redis cache in front of postgres” insert, inference Rename “Rename api to checkout” update Scale a service “Scale api to 4 instances” update Undo by voice “Add a Redis cache” → “Undo that” history Redo by voice “Add Kafka” → “Undo” → “Redo” history Remove an unconnected component “Add a scheduler” → “Remove the scheduler” remove Destructive removal asks first “Delete the api” remove, confirmation Plaintext link, then encrypt it “Add a web app and a load balancer” → “Encrypt the connection from web to lb” connect, security Simulate 10x traffic “Simulate 10x traffic” simulation Kill the database, then fix it “Kill orders-db” → “Fix it” simulation, chaos Survive a 10x surge “Simulate 10x traffic” → “Fix it” simulation, remedy Fix the single point of failure “Fix the single point of failure on orders-db” linter Load a template “Load the realtime chat template” canvas Worker consumes a queue “Add an SQS queue” → “Add a worker that consumes from the queue” add, connect Gateway in front of the API “Put an API gateway in front of the api” insert