How to evaluatea studio like ours
Evaluating an agent systems studio comes down to seven questions: testing, approval, model changes, failures, who builds it, ownership, and confidentiality.
We wrote this checklist, and we are one of the studios it evaluates. Our answers include the limits. Use it with every studio you consider.
Seven questions
Question 01
Can you show me how you test the system?
Check whether the demo has repeatable tests and clear rules for what the agent does next.
Look for
They run the tests and show you the evaluation cases and decision rules in the repository.
Walk away
They only replay the demo or describe a prompt without showing how they check its behaviour.
Cascadia’s answer
Our concierge has a published architecture. Our outbound rules run as automated checks with a test suite.
Question 02
Where does it stop for a person?
Identify which actions need approval and who sets those boundaries.
Look for
They name the first irreversible action and place approval before it. They explain what runs unattended.
Walk away
They offer unrestricted autonomy or notify you only after the action has happened.
Cascadia’s answer
We require approval before the first irreversible action. Work upstream runs unattended; guardrail failures block the action.
Question 03
What happens when the model changes?
Check whether the system can use a different model without losing the behaviour your team relies on.
Look for
They explain how to replace the model and run evaluations that reveal regressions.
Walk away
They cannot explain how decision rules survive a model change or how they would test it.
Cascadia’s answer
The model is replaceable. Our decision rules run in code and can be tested without a model or API key.
Question 04
Show me something that did not work.
Ask for a specific failure and the change made to prevent it recurring.
Look for
They describe what failed, what it cost, and what changed. They can show the fix.
Walk away
They offer a disguised success story or blame the client without explaining a fix.
Cascadia’s answer
Generated text broke a documented rule twice. We replaced the rule with a test that fails the build and published the case study.
Question 05
Who will write the code?
Meet the person responsible for building the system before you commit.
Look for
The builder is on the call and answers technical questions about your project directly.
Walk away
They describe a team but cannot identify who will build your system or let you speak with them.
Cascadia’s answer
One person scopes, builds, and stays accountable, supported by agents. We are not the right fit for work requiring six people in parallel.
Question 06
What do I own when the work ends?
Confirm where the code, infrastructure, and documentation live, and what depends on ongoing support.
Look for
The system runs in your accounts. Another engineer can use the documentation, and support terms are clear.
Walk away
The system depends on their platform, or they defer the ownership terms until the project ends.
Cascadia’s answer
You own the repository, infrastructure, guardrails, and evaluation set in your accounts. Ending our engagement stops our monitoring; the system keeps running.
Question 07
How do you discuss confidential work?
Check whether they can explain relevant work while respecting the client’s confidentiality.
Look for
They explain the industry, problem, architecture, and type of outcome without identifying the client.
Walk away
They share a confidential client name or identifying detail to impress you.
Cascadia’s answer
Two current engagements are confidential. We describe them by type. Automated checks block drafts that contain protected names.
Check ours
Inspect the systems we use ourselves, including their limitations.
Bring the list.We will answer all seven.
Bring your questions and a real problem. We will show you how we would approach both.