I Built a Real-Time Voice Coach for Age of Empires 2

Work in progress. I have better videos and demos than the ones in here. I wanted to start sharing this now anyway.

Age of Empires 2 is a real-time strategy game. You start with a handful of villagers, gather wood and food and gold, raise a town, advance through a series of ages, and build an army to overrun your opponent before they overrun you — all at the same time, with a clock that never stops and far more to keep track of than any one person comfortably can. Hold that picture for a second.

It's minute eleven of a ranked match. I'm scrambling to advance into Castle Age, my scout has just bumped into the enemy's town, and I've completely failed to notice that two of my villagers have been standing next to a tree doing nothing for the last thirty seconds. Then a calm voice in my headphones says:

"Two idle villagers — put them back on wood until you have 12. And you've found him: he's massing archers, so start thinking skirmishers."

I didn't type anything. I didn't pause the game. I didn't ask. The voice just noticed, the way a friend watching over your shoulder on a Discord call would.

That voice is named TrebuCoach — a trebuchet (Age of Empires 2's most gloriously over-discussed siege weapon) welded to a coach — and I built it.

A little while back I wrote about spending a weekend building a macOS driver so I could fly Microsoft Flight Simulator with a borrowed joystick. This is the next entry in what is apparently becoming a series: Leo gets obsessed with a game and builds something slightly unreasonable to play it better. I'm a software engineer and CTO at Pento AI, I build AI products for a living, and on weekends I point that same energy at whatever's currently eating my evenings. Right now, that's a 25-year-old strategy game.

A quick word on Age of Empires 2

If you've never touched it, here's the fuller picture. It's a game from 1999 that simply refused to die. It got a Definitive Edition in 2019, a thriving competitive ladder, tournaments with big prize money, and a community of players who treat a game older than most of their teammates like a living sport.

The premise is deceptively simple. You start with a town center and a handful of villagers. You gather wood, food, gold, and stone; you grow your economy; you advance through four Ages — Dark, Feudal, Castle, Imperial — and somewhere in there you build an army and go kill the other player before they kill you. It's chess where both sides move at once, the clock never stops, and you're also responsible for farming the wheat.

The part that hooks you is the sheer number of things you're supposed to be doing at the same time. Keep every villager working. Keep your town center producing. Don't get "housed" (capped on population because you forgot to build houses). Don't let resources pile up unspent. Scout the map. React to what the enemy is doing. Hit each Age on time. Miss any one of these for thirty seconds and you've quietly handed your opponent an advantage you'll spend the next ten minutes paying for.

The boring truth about getting better

Here's the thing I've come to believe after far too many ladder games: the difference between a 900-elo player and a 1400 usually isn't dazzling micro. It's the boring stuff.

It's never letting the town center sit idle. It's not floating 600 wood while you're stuck under the population cap. It's hitting Feudal at a reasonable time instead of two minutes late. It's reacting when your scout finds the enemy instead of finding out the hard way. None of that is hard to understand. It's hard to remember — all of it, continuously, while forty other things are also demanding your attention.

Which is exactly the kind of job a coach is good at. A strong player watching your stream wouldn't need to play for you or out-think the game tactically. Most of the time they'd just remind you, at the right moment: "town center's idle," "you're floating gold," "go up to Castle." Gentle nudges, well-timed.

And that — a thing that watches your screen and reminds you of the obvious at the exact moment you've forgotten it — felt narrow enough to actually build.

The catch: the game won't tell you anything

There's one wrinkle. Age of Empires 2 has no public, real-time API for live matches. There's no tidy data feed I could subscribe to that says "wood: 540, idle villagers: 2, age: Feudal." For a live game, that information exists in exactly one place: on the screen, in the HUD, the same pixels you're looking at.

So I decided to read it the way a human does. Look at the screen. Understand what's happening. Say something useful out loud.

Spelled out, the coach needs three abilities:

1. See the screen — specifically, read the numbers in the HUD.
2. Notice when something coaching-worthy changes — idle town center, floating resources, an Age-up, the enemy getting scouted.
3. Say something short and useful, out loud, without being annoying.

Three abilities, three pieces of technology. And here's the part I find genuinely fun: I used AI to build an AI. The whole project is really just the wiring between three off-the-shelf superpowers — eyes, a voice, and a bit of judgment — none of which I had to build from scratch.

The three pieces

Eyes — a vision model. Since the only source of truth is the screen, I send a screenshot to a multimodal model (I use Google's Gemini, with the option to route through OpenRouter to try others) and ask it to read the HUD. The trick that makes this reliable is structured output: instead of a chatty paragraph, the model hands back a clean, fixed-shape JSON reading — resources, population, current Age, idle villagers, whether the enemy is in view. A picture goes in, structured data comes out.

A voice — ElevenLabs. This is the heart of it, and the part I expected to be hardest. ElevenLabs Conversational AI isn't just text-to-speech; it's a full voice agent you talk to in real time over a connection. It has a persona, it listens to your microphone, it talks back, and — crucially for me — it can quietly receive updates in the background and weave them into a natural spoken reply. That last capability is the whole magic trick, and I'll come back to it.

A body — Tauri. All of this needs to live in a real desktop app that can take a screenshot of a specific window, hold my API keys somewhere safe, and listen for a global hotkey even while Age of Empires has my full attention. Tauri gives me a small native app with a Rust core for the privileged work and a plain web UI on top. Small, fast, and it keeps my secret keys well away from the part of the app that talks to the internet.

Put together, a single moment of a live game looks like this:

Your Age of Empires 2 window
  → screenshot, every ~5 seconds (Tauri / Rust, off in the background)
    → shrink it, send to the vision model (Gemini)
      → a structured reading of the HUD (resources, Age, idle vills, enemy in view)
        → compare to the last reading: anything worth mentioning?
          → ElevenLabs voice agent (a silent feed of the game + the occasional nudge)
            → 🔊 TrebuCoach says one short, useful thing

The trick that makes it feel like a coach

When most people picture "talking to an AI," they picture a chatbot — you type, it answers, repeat. TrebuCoach isn't that. I almost never talk to it. Instead, every few seconds the app slips the agent a silent, behind-the-scenes update describing what's currently on my screen, plus the occasional little poke like "the player just got housed." The agent folds all of that into the conversation and decides whether it's worth speaking up.

The effect is that TrebuCoach feels less like a chatbot I'm interrogating and more like a friend who's been quietly watching my feed the whole time and pipes up exactly when it matters. I never had to build the genuinely hard parts of a real-time voice experience — knowing when to talk, handling me interrupting it, turning text into a believable voice. ElevenLabs handles all of that. I just feed it a good picture of the game and stay out of the way.

The other half of "feels like a coach" is restraint. A coach that narrates everything is unbearable. So I added a verbosity setting — at the quiet end, TrebuCoach only opens its mouth for things that genuinely matter — and a few rules so it won't talk over me or repeat itself. The goal was never a play-by-play announcer. It was the friend who stays mostly quiet and then says the one thing you needed to hear.

Setting up the agent was the easy part

I want to be honest about the part that surprised me most: standing up the actual voice agent was easy. I'd braced for the hardest engineering of the project — and it was mostly filling out a form.

In the ElevenLabs dashboard, building TrebuCoach came down to a handful of decisions:

• A system prompt that gives it a personality and teaches it the conventions of my little world — which incoming messages are silent context to read but never speak, which are nudges that deserve a short reply.
• A knowledge base so it actually understands Age of Empires 2 — build orders, civilization bonuses, what each unit is for.
• Guardrails to keep it on-topic and out of trouble.
• Picking the model behind the conversation.
• Picking a voice — the part where TrebuCoach stopped being an abstraction and started being a guy.
• And then connecting to it from my app over the real-time API.

That's it. The thing I assumed would be weeks of fiddling with speech synthesis and conversation timing was an afternoon of configuration. Most of my actual engineering time went into the eyes — reading the screen reliably — not the voice.

Teaching it to see (without going broke)

The vision side took the most care, and a couple of decisions made it work.

The first was teaching the model to admit when it can't read something. A screenshot during a chaotic fight, or while the camera's halfway across the map, won't always show a clean resource count. The instinct of a language model is to guess a plausible number. That's poison for a coach — a hallucinated "you're floating 800 gold" makes TrebuCoach sound confident and wrong. So the rule baked into everything is: when you can't read a value, say so, return nothing, don't invent it. Missing numbers get carried over from the last good reading instead of flickering to zero. The result is a game state that's honest rather than noisy.

The second was being able to watch what the model sees without paying for the voice. I built the coach as three switches rather than one big "start" button: capture the screen, read it with vision, and connect the agent — each one independently on or off. With the first two on and the agent off, a live log shows me every reading the model makes and the exact sentence it would have spoken, tagged as a preview. I could tune the eyes for an entire match without spending a single second of ElevenLabs voice time. That one decision is the biggest reason the thing got good.

Screenshot: the pipeline log showing live vision readings and the lines TrebuCoach would speak, in preview mode with the voice agent switched off Screenshot: the pipeline log showing live vision readings and the lines TrebuCoach would speak, in preview mode with the voice agent switched off

The third was just... not being wasteful. Full-resolution screenshots are slow to send and expensive to analyze, and on a five-second loop that adds up fast. Shrinking each frame to half size before it goes to the model keeps the HUD numbers perfectly legible while cutting the data roughly four-fold. Cheaper, faster, just as accurate.

Does it actually work?

Mostly, yes — and using it has been more fun than I expected.

It knows the build order I told it I'm running, and right out of the gate it points me at the economy and the things I should be watching for. When I reach a new Age, it walks me through what my economy should look like and how to actually use the units I've just unlocked — and it pays attention to both the small, fiddly actions and the big-picture macro at the same time. The moment my scout reveals the enemy, it understands what I'm looking at and starts suggesting how to respond. And it's most useful in the exact spot where I'm usually worst: that mid-game moment where you're a little lost, or you can feel your plan falling apart and you need to switch strategies.

When I do want to ask it something, I just hold a button to talk:

It's not perfect. This is the first version, and sometimes TrebuCoach says things that are flatly wrong — misreads a number, or gives advice that doesn't fit the moment. I'd rather tell you that than pretend I shipped a grandmaster in a box. But the floor is already useful, and the boring habits it reinforces are exactly the ones that quietly win games.

What it can — and can't — see

There's an honest edge to this whole approach, and I think it's the most interesting part.

Because TrebuCoach only knows what's on the screen, it's great at everything the HUD shows: floating resources, getting housed, Age-ups, idle villagers, hitting villager milestones, spotting the enemy. What it can't see is anything the game doesn't draw. There's no on-screen counter that says "you're on step 7 of your Scouts rush," so it can't automatically track how far through a build order you are — it knows you've reached Feudal, but not that you were supposed to get there ninety seconds ago.

That boundary is the whole philosophy of the project, stated plainly: the coach knows exactly what a careful human could read off your screen, and nothing more. Which, for the unglamorous, decisive habits that actually decide most games, turns out to be plenty.

What I keep thinking about

The same theme that ran through the flight-stick weekend runs through this one, just from a different angle. That project was about how AI collapses the ramp-up cost of an unfamiliar domain. This one is about something I think is bigger.

A few years ago, "an app that watches your game, understands it, and coaches you out loud in a natural voice" would have been a research project — a team, a budget, papers. Today it's three capabilities you can rent by the API call, and the actual work is taste: deciding what to wire to what, when the coach should speak, what it should ignore, how to keep it honest. The hard part has moved up the stack, from can this even be built to should it say this, right now.

And there's a quieter shift underneath it. We're used to AI as a thing you go to — a tab you open, a box you type in. TrebuCoach is AI that sits beside you, watching the same screen you are, speaking up only when it's worth it. That's a different relationship with a machine, and the first time a voice in my headphones caught an idle villager I'd missed, it stopped feeling like a tool and started feeling like company.

The plane flew. Now the coach talks. I have no idea what the next game I get obsessed with will be — but I'm fairly sure I'll end up building something faintly ridiculous to play it better.

The project is called Eleven AoE, and the code is on GitHub. A few things I left out for readability, and a few rough edges I'm still smoothing.

A few things I'd like to try next:

1. Track build-order progress against the in-game clock, so TrebuCoach can tell you you're behind your own plan, not just where you are.
2. A proper on-screen overlay, so the coaching lives on top of the game instead of beside it.
3. More voices and personalities — a calm coach for ladder anxiety, and a far less calm one for when you're throwing a winning position.