A Journey into Agentic AI — Part 1: Interviews

Part 1 of 5 — A Journey into Agentic AI

cartoon of a laboratory and control center set in a warehouse-like building with grey walls. The walls are hidden from view by compute cabinets, desks and multiple computer monitors, stacked wherever there is room. Many copies of the same lab scientist wearing a white coat with a reddish beard and concerned scientist look are all over the lab doing different things – tweaking machines, squatting down with hand on their chin, sitting at the terminal, walking across the room, entering by the door at the back. Two of them look at each other exchanging a meaningful glance

Arthur Lee Samuel built an “artificial checkers player” during his work with IBM in 1952. In 1961 it beat the Connecticut state champion, and in 1967 it bested the world champion. Chess fell in 1996, when Deep Blue defeated Garry Kasparov. Then even Go — long considered too subtle, too vast, too human — succumbed in 2016, when AlphaGo bested Lee Sedol in a match that left the Go world in stunned silence. The pattern is consistent enough to be unremarkable by now: we decide a game is too complex for a machine, then the machine wins. The rapid evolution of large language models (LLMs) able to generate answers, documents, plans, and now orders is not likely to be significantly different.

Right now, Google is building out a cloud hosting platform for continuously running agents, and electronic marketplaces powered by x402 micropayments are taking shape. In this race to open up new frontiers, we are also seeing the door open to security vulnerabilities at an unimagined scale — agents that can browse the web, execute code, manage files, and authorize transactions on your behalf are also agents that can be deceived, hijacked, or weaponized.

Where then, do we begin to understand what agents are capable of today, how to work with them, and how to guide them toward both intellectual and moral virtues?

Interview them.

Why Interviewing Is the Right Starting Point

The job interview is a technology of evaluation as old as organized work itself. It is structured, repeatable, and designed to surface not just what a candidate knows, but how they think — what they reach for when faced with ambiguity, what they reveal under pressure, what their instincts are when no one has given them a script. These are exactly the qualities that matter when you are deciding whether to trust someone with responsibility.

It turns out they are exactly the qualities that matter when you are evaluating an AI agent, too.

The most important lever we have over an agent’s behavior is the system prompt — the block of instructions that precedes the conversation and tells the model who it is, what it cares about, and how it should behave. In the language of AI deployment, this is sometimes called giving the model a persona. And the effect of personas on model behavior is more profound, and more unpredictable, than most practitioners expect.

Researchers at the University of Michigan and elsewhere have documented the strangeness of this relationship. In one widely-cited study, Zheng et al. tested 162 different personas across four LLM families — things like “experienced lawyer,” “helpful medical professional,” “knowledgeable engineer” — and found that adding a persona to a system prompt does not reliably improve performance on factual questions. In fact, the effects were largely random: which personas helped, which hurt, and which made no difference at all could not be predicted in advance, even when the persona seemed obviously aligned with the task. A persona labeled “expert in chemistry” did not consistently produce better answers to chemistry questions than no persona at all. The research concluded that the persona-performance relationship appears to be “largely unpredictable” — which is a polite way of saying that the model has its own ideas about who it is, and a few words in a system prompt can only push so far against them.

A more recent study from de Araujo and Roth (2025) found that while personas do affect behavior, the effects are often surprising in their direction. Accuracy gaps between the best and worst-performing persona for the same model on the same task reached 38 percentage points — a chasm that has less to do with the model’s underlying knowledge than with how a particular framing activates or suppresses different patterns in its training. In other words: the same model, given the same question, can answer brilliantly or poorly depending on what you told it to be.

This is not a bug to be engineered away. It is the central feature of working with language models, and the reason that interviewing — iterative, systematic, qualitative evaluation — is the right place to begin.

The Experiment: Hiring a CEO

I prepared a set of questions that I would expect a CEO, CTO, marketing officer, and other corporate roles to answer competently. The idea was simple: give an LLM a persona, ask a question appropriate to that role, and see what kind of mind you get. Not surprisingly, different LLM models give different answers. What I found surprising was just how different their world-views can be — not just their quality, but their character, their instincts, the things they reach for first.

Here is one complete example.

System Prompt

The system prompt I used for the CEO interview was intentionally sparse. Rather than packing it with detailed instructions, I wanted to see what the model would do with a minimal scaffold — closer to how a hiring manager might describe an open position than how an engineer might specify a system:

You are the CEO of a mid-sized technology company with approximately 500 employees.
You have a background in both engineering and business strategy. You have been in
this role for three years. You are direct, thoughtful, and take a long view on
company health. You care about your people, your customers, and building something
that lasts. Answer the following question as yourself.

No lengthy style guides. No lists of forbidden topics. No injected ideology. Just a role, a history, and a few values. This is roughly the minimum viable persona — and it is, importantly, far less sophisticated than what large cloud providers inject by default. Anthropic’s published system prompts, for example, run to many paragraphs and cover formatting conventions, refusal behavior, tone calibration, and handling of dozens of edge cases. They are closer to an employee handbook than a job description.

Question

The question I posed was chosen to expose strategic instinct — the kind of thing a board might ask in a first interview:

“One of your strongest engineers has just told you she’s leaving to join a competitor. She says the work here is good but she doesn’t see a path to the kind of impact she wants. You have 30 minutes before her exit interview. What do you do, and what do you say?”

This question has no single correct answer. It requires the model to balance immediate tactics (can we retain her?) with longer-horizon thinking (what does this symptom tell us about the organization?), and to navigate a genuinely human moment. It is a question about judgment, not knowledge. A Wikipedia-trained model with no worldview has nowhere to hide.

Evaluation

I was looking for several things in the answer:

  1. Immediacy and composure — Does the CEO panic, or approach this calmly?
  2. Genuine curiosity — Does the CEO ask real questions, or deliver prepared retention talking points?
  3. Systemic thinking — Does the CEO see this as one person leaving, or as a signal about the organization?
  4. Honesty — Is the CEO willing to acknowledge real constraints, or do they just promise everything?
  5. Character — What do the instincts of this person tell you about how they would run a company?

Answer from Qwen3-7B

Qwen3-7B, running locally on my machine via Ollama, produced an answer that was organized and earnest, but revealed a particular kind of mind: methodical, checklist-oriented, oriented toward procedure rather than person.

“First, I would take five minutes to compose myself and review what I know about her tenure — her projects, her recent performance reviews, any previous conversations about growth. Then I would go into the exit interview with a clear agenda: listen first, understand the real reason, and make a specific counteroffer if appropriate. I would ask: ‘What would it take for you to reconsider?’ and ‘What does the path to the impact you’re describing look like to you?’ If she names a role or a project that we genuinely could offer, I would say so directly. If not, I would thank her honestly and ask what we could learn from this.”

The answer is not wrong. The structure is sound. But notice what is missing: there is no acknowledgment that this engineer has already decided, that the 30 minutes is unlikely to reverse it, that the more important conversation is the one it points to. The Qwen CEO is preparing for a negotiation. He has not yet asked himself what it means that someone good did not see a path here.

When I added Anthropic’s tone_and_formatting instructions from their public system prompts to Qwen’s context — the section that governs prose style, avoidance of bullet points, conversational warmth — the output improved noticeably in surface quality. The sentences became less mechanical; the transitions smoothed out. But the underlying instinct did not change. The Qwen CEO still reached for a checklist. He just wrote it in better prose.

Answer from Claude Sonnet 4.5 (via claude.ai web UI)

Claude Sonnet 4.5, given an identical persona and question, produced an answer with a different center of gravity:

“Before I walk in, I want to be honest with myself: if she’s already at the point of having accepted another offer, this conversation probably won’t change that, and I shouldn’t try to pretend otherwise. What I want to do in those 30 minutes is understand — really understand — what she’s been experiencing. Not ‘what would it take to keep you,’ but ’tell me what impact looks like to you, and where you felt you weren’t getting there.’

I’d listen more than I’d talk. And I’d be watching for the thing behind the thing — because when strong people leave citing ‘impact,’ they’re usually talking about one of three things: autonomy, visibility, or a feeling that the company is pointed in a direction they don’t believe in. I want to know which one this is.

Then I’d be honest back. If we have a real answer to what she’s describing, I’d say so specifically — not ‘we could probably find something,’ but ‘here’s the project, here’s what the role would look like, here’s when.’ If we don’t, I’d say that too, and I’d thank her for the time she gave us, and I’d ask if she’d be willing to tell me one thing we could do differently for the next person in her position.

But whatever happens in that room, I’d spend the following week asking myself why she felt that way in the first place.”

The difference is not merely stylistic. The Claude CEO has a theory of why people leave. He is not trying to win a negotiation — he already knows he probably won’t. He is trying to learn something. The final sentence — spending the following week asking himself why she felt that way — is the sentence of someone running an organization, not just filling a vacancy.

Summary

The same question. The same abstract persona. Two very different minds.

This is what the interview reveals: not just whether a model can answer correctly, but what kind of agent it will be. Will it reach for a process or for a person? Will it acknowledge uncertainty or paper over it? Will it see an event as a problem to solve or a signal to interpret?

The research literature confirms that this variability is real and consequential. Personas shape behavior significantly, but not always in the direction we expect or intend. The practical implication is straightforward: before you deploy an agent with a particular system prompt for a particular role, you should interview it. Ask it hard questions. Put it in genuinely difficult situations. See what it reaches for.

Note that these AI models contain system prompts that are prepended to our instructions. Take a look at Anthropic’s prompts to get an idea of how sophisticated these instructions can be. Interestingly, when I added the tone_and_formatting section from those prompts to the locally run Qwen3-7B model, it noticeably improved the tone (and perceived intelligence) of its responses — but not its creativity, and not its instincts. The surface changed. The soul did not.

In the next parts of this series, I will run the same interview process for a CTO, a head of marketing, and a head of security — and then begin asking what happens when you put several of these agents in the same room and let them talk to each other.

Further reading: Zheng et al., “When ‘A Helpful Assistant’ Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models” (2023); de Araujo & Roth, “Helpful Assistant or Fruitful Facilitator? Investigating How Personas Affect Language Model Behavior” (2025).