As much as it can do without a person needing to be in the room, and it should be the one to notice when that stops being true. The number of tasks an agent completes on its own tells you very little. What matters is whether it stops and asks at the right moments, and I think that is something you can design for rather than hope for.
What did the factory ask me when it merged 18 feature flags?
I build multi-agent systems for a living. A good deal of Genie's engineering now goes through what we call the Software Factory, agents that build and review production code under rules a small team of us wrote in advance. In October I gave it a job I'd usually spread across a week: merge 18 feature flags into a major Genie release and review the result.
It started one afternoon and was still going the next morning. Over that time it came back to me three or four times. Each question was one its escalation rules said it had to ask. Everything else it decided on its own.
"Everything else" covers more than it sounds. Whether a flag needed breaking into smaller pieces. Whether a change needed a second agent to look at it before it was merged. Which tests to run, and whether a failing one was telling the truth. None of that came to me, and I didn't want it to.
I was more interested in the three or four than in the 18. It would have been easy to build something that checked in constantly, and easy to build something that never did. Neither would be much use. What I wanted was an agent that asked about the handful of things I'd told it were mine, and got on with the rest.
Who decides what goes on the list?
I did, before the first job ran. The agents have a defined scope. Three things always come back to me: cost, architectural decisions and legal quality. If a piece of work touches one of those, it stops and waits. If it doesn't, it carries on.
There's a rule for when I'm not around too. After 9pm the factory switches to what I called night mode: I'm not available, so it makes the decisions it can inside its scope and leaves the rest for me in the morning.
It took me a while to get here, to be honest. My first attempts looked a lot like an org chart for software, with phases, gates, checklists and escalation paths borrowed from how we manage people. I'm less convinced by most of that now. The escalation list is the part I'd defend.
It also holds up when the job gets bigger. The technology under Agentic Legal Specialists was built by 75 agents working through an afternoon and overnight, and that run came back to an engineer four or five times. Roughly the same handful of questions, on a job many times the size. That's the behaviour I'd point to if someone asked me whether the factory had learned anything.
Why does this matter more for legal work than for code?
When an engineering agent gets an autonomous decision wrong, we roll it back. When a legal agent gets one wrong, a clause has been agreed that nobody read, and the person accountable for it is a lawyer, not a model. So the discipline the factory runs on matters more here, not less. The agent needs to know what it can settle alone and what has to go to a named person, and that rule has to be enforced somewhere firmer than the prompt.
I say firmer than the prompt because I've seen what happens otherwise. Last year our legal team found two failures in a single real conversation with an earlier version of our document agent. The summary said a contractual right was mutual, and the clause it cited gave that right to one party only. A passage shown to the user as a direct quote wasn't a quote at all, but a rewritten version of the clause inside quotation marks. Nothing errored. The answers looked fluent and complete. The prompt had rules against both, and the rules failed silently. That is the kind of decision an agent should never be settling on its own, and the only way to be sure it doesn't is to check the output somewhere the generating model can't reach.
That's what we have built into Agentic Legal Specialists. For each piece of work the agents decide which specialists the job needs and which checks must pass before anything leaves. Routine steps run inside the customer's own rules. Exceptions go to a person. You can only move as fast as you can trust, and in my experience trust comes from drawing that line before the work starts, with an agent that can tell which side of it a decision falls on.
What should a legal or commercial team take from this?
Before asking how much an agent can do, I'd write down what it's allowed to settle on its own, what has to come to a named person, and what happens out of hours. Then ask the vendor to show you where that line sits in their system, and what happens when a decision lands on the wrong side of it. Accuracy percentages are easy to quote. What you want to know is what the system does when it's wrong, and whether anyone finds out. If they can't answer that, the amount of autonomy on offer isn't the useful number.
If you'd like to see where it sits in Genie, on one of your own contracts, book a demo and we'll run it with you.
Frequently asked questions
Does giving an AI agent more autonomy mean less control for the legal team?
Giving an AI agent more autonomy does not reduce the legal team's control, as long as the escalation rules are written down first. Control sits in those rules, not in watching every step. When our software factory merged 18 feature flags, it asked me three or four questions in total, and every one was a decision I would have wanted.
What happens when a software factory agent is unsure whether to proceed?
A software factory agent that is unsure stops and asks rather than guessing. In our factory, anything touching cost, architecture or legal quality lands with me, and after 9pm it holds those questions for the morning. In Genie, anything outside the standards the customer has set goes to a named person instead of being settled by the agent.
Can you change an AI agent's escalation rules once it is running?
Yes, an AI agent's escalation rules can be changed while it is running. In our software factory the scope is a document the agents read before each job, not something buried in code, and we revise it as we learn what they get right on their own. A customer's standards in Genie work the same way.
