# Why we hired the AI agent that came third

> In eight of nineteen hires, our top-scoring AI agent didn't get the job. Why we keep the full record of every hire, and what to ask a legal AI vendor.

**Author:** Daniele Tassone  
**Category:** Insights  
**Published:** 2026-11-20  
**Reading time:** 5 min

If an AI gives you an answer and can't show how it got there, all you can do is trust it or not. We don't accept that from the agents that build Genie. For every one of them we keep the full record of how it was chosen and what it beat. That record often disagrees with the top score. Here's one hire where it did, what we keep each time and what it means for any AI you let near a contract.

## The hire that came third on paper

Much of Genie's engineering runs through [the Software Factory](https://www.genieai.co/blog/how-much-should-ai-do-on-its-own). We don't write the agents in it by hand. Each one is hired: for every role the factory generates a wide field of candidate profiles, 45 in the current version, and screens them down to the five who sit an exam.

In September we hired a product design agent this way. The five candidates were interviewed for real against the same exam and two judges scored every answer blind, using marking criteria the candidates never saw. Out of ten, they landed between 9.0 and 9.4.

The one we hired scored 9.2. It came third.

A gap that small is judge noise, so the top three went head to head. Our eventual hire won the head-to-head round and the two with higher scores didn't. There was still one more stage. It had to pass a task we'd held back from the exam, marked by a judge who hadn't seen the earlier rounds, and it passed at 8.7. The judge took marks off for one thing: it had called a diagram mandatory and then hadn't produced one. That went into its file as the first thing to coach.

What each stage of the hire could tell apart. Stage one, the exam: five candidates scored 9.4, 9.3, 9.2, 9.15 and 9.0 out of 10, too close to separate. Stage two, head to head: the top two scorers lost and the third-placed candidate won. Stage three, a task held back from the exam: the third-placed candidate scored 8.7 out of 10 and passed, with one small miss, a diagram it left out.

## Why a score isn't working

With only the leaderboard to go on, we'd have hired a different agent and never known. A score tells you who came top. It says very little about whether to believe it.

It happens more often than we expected. In eight of our nineteen hires so far, the candidate with the highest exam score didn't get the job. Most of them lost the head-to-head and one failed the held-back task outright.

Human recruiting spent a century learning that gut feel doesn't work and that the person who interviews best isn't always the one who does the job best. Agents make this harder. As I wrote [last time](https://www.genieai.co/blog/smarter-agents-find-better-shortcuts), a capable model produces work that reads as finished whether or not it is, so you should expect five candidates who all sound excellent. You only see the differences when you make them do the task and put the results side by side.

## What we keep from every hire

Every stage leaves a record and we keep all of it. There's the brief, with the requirements the hire has to meet and where each one came from. There's the exam, both judges' scores for every candidate, the duels and the verdict on the held-back task. For any agent in the factory we can show you why it was chosen and what it beat.

How an agent is hired and what is kept. A brief lists requirements, each with a source. 45 candidates are generated, five sit the exam marked blind by two judges, and three go head to head. The winner must pass a task held back from the exam. If it fails it is rejected and the next finalist is tried. Every stage is kept on record.
We needed that record in July. The agent that won our backend role turned out to have lost concrete knowledge of our codebase that the agent it replaced had carried. The record showed us where it had gone. The brief hadn't asked for it, so the exam didn't test it and the judges had no reason to mark it down. We put the knowledge back by hand and then changed the process, so every hire is now checked line by line against its requirements before it starts work.

It isn't finished. A record only shows you what was asked. One of our hires passed every stage and still fell short in real work because the exam had never tested the skill that turned out to matter. We ran that hire again and we now review each new one against real work once it has started.

## The same test for legal work

A senior lawyer who gets a conclusion from an associate with no reasoning sends it back. We think an AI should get the same treatment. It's harder to give, though, because an AI's answer turns up as clean, confident text whether the reasoning was sound or not.

So Genie's edits arrive as tracked changes. [The Agentic Editor](https://www.genieai.co/product/agentic-editor) works in your original Word file and writes each change as a native tracked change with an explanation. You can accept it or reject it. Whatever it didn't touch stays as you sent it.

Law is a regulated field. When someone who isn't an engineer asks why the system did what it did, the record is the answer you can give them. We want that to be true of the agents that build the product and of the work the product hands you.

## What to ask any legal AI vendor

When a vendor tells you their AI is accurate, you should ask what it was chosen over and whether you can see the comparison. We'd also ask to see the working on a single answer, so you can tell where each point came from in the document and which changes a person can reject. And if the AI turns out to be wrong, you'll want to know what the record shows. A vendor who can only offer the final answer and a score is asking you to take the AI's word for it. We'd say the same about us.

If you'd like to see the working on one of your own contracts, [book a demo](https://www.genieai.co/book-a-demo) and we'll go through it with you.

## Frequently asked questions

### What does it mean for an AI to show its working?

It means the answer comes with its evidence. You can see where each point came from, what was changed and what was considered and turned down. In Genie that's a tracked change with an explanation in your own Word file. In our engineering it's the full record of how each agent was hired.

### Why not just hire the agent with the highest score?

Close scores sit inside the judges' margin of error, so the ranking between them doesn't mean much. In our hiring, candidates within a few tenths of a point go head to head and the winner still has to pass a task it hasn't seen. In eight of nineteen hires so far, the top scorer didn't get the job.

---

This is the Markdown representation of [https://www.genieai.co/blog/why-we-hired-the-ai-agent-that-came-third](https://www.genieai.co/blog/why-we-hired-the-ai-agent-that-came-third), provided for AI agents and crawlers. The HTML page is canonical. See [/llms.txt](https://www.genieai.co/llms.txt) for the full content map.
