What I Learned Building Carla and Andrea, My Two AI Agents

I built two production AI agents for a client in the Italian real-estate auctions space. Carla answers the phone. Andrea writes on WhatsApp, email, and the website chat. Carla talks, Andrea writes, and together they try to move interested people toward a real consultation with a human consultant.

Carla went live on 2 September 2026. Andrea came a few weeks later, around 20–23 September, because the same “I’m interested” message was arriving dozens of times during the night and nobody answered until the morning. Automation does not start because you want AI. It starts because something stupid happens again and again, and at some point you say, okay, enough.

Since then I changed them constantly. In September alone, the changelog has 134 entries about the agents. In October, 62 by the 20th. Carla’s prompt files changed in 30 commits, Andrea’s in 15, and the internal diary literally says the agents “change almost every day”. So yes, this is not the nice demo where the agent books a table in a restaurant and everybody claps. This is production. Production is a mess.

1. Build Evals Early, Because You Will Retest Everything

Build your testing and eval framework as soon as possible, because you will need to retest everything more often than your optimism wants to admit. Carla started on Gemini 3 Flash preview. On 10 September I benchmarked 8 models and one “lite” model looked amazing: first word in 0.66 seconds in tests. Same evening I reverted it, because live on the phone the first word went from around 1 second to more than 2 seconds, and at some point the model started reading tool-call code out loud to the caller.

The big problem with voice is that a written judge does not hear what is actually spoken. It reads text. The caller hears pauses, interruptions, weird phrasing, and sometimes things that should never be said aloud. So Carla has 30 inbound and 13 outbound scenarios, every output passing through a spoken-text filter, an unsourced-number detector, and a merit check.

And even the judge breaks. On 1 October the evaluator scored every scenario 0 because the platform changed the wording of its verdict string. Since 13 October I also have an hourly judge: a Claude Code session reading conversations against hard limits, giving each defect a colour per round. I wrote about the judge separately, but the short version is: it catches what I would miss, like the time Andrea started booking times the person had not chosen, in 4 of 54 threads.

The funny part is Luna. On 6 October I rejected GPT-6 Luna on ElevenLabs because it scored 25 out of 36 and was missing urgencies. On 16 October I retested it and suddenly it scored 33 out of 36, with 0 empty promises and 9 out of 9 urgencies, at about a tenth of the cost. We put it live on Andrea, and over nine full days credits per reply went from about 177 to about 42.5, minus 76%.

But on Carla it failed badly. It hung up before the caller said goodbye in 11–12 of 30 scenarios, and the judge missed it. Same model, same platform, totally different result. Welcome to agents.

2. Shorter Prompts Work As Well, Sometimes Better

Do not assume a longer prompt is a safer prompt, because in production it can be slower, more expensive, and not even more accurate. On 16 October I tried Claude Haiku 4.5 on Carla from 13:31 to 14:50. First word was 0.54 seconds, which is beautiful on paper. But 3 calls out of 15 were lost: one with “All LLMs have failed” because there was no fallback model, and two where Haiku called the “wait” tool right after the caller answered, then the line closed after 30 seconds of silence.

It also cost twice as much per minute as Gemini 3.6. Live, it rewrote the whole long prompt into the cache 5–6 times per call. We had decided the rollback rule beforehand, so at 14:50 Carla went back to Gemini 3.6.

On 19 October, ElevenLabs support wrote that Carla’s prompt was too big and should come down to around 10,000 tokens. Long prompts perform worse and trigger the fallback model more often. So on 21 October I built short Carla: prompt from about 89,000 to about 31,000 characters, knowledge retrieved each turn from 50,000 to 8,000 characters.

The trick was not to remove the rules. Same rules, approved sentences word for word, same tools and parameters. Tool descriptions only say when to call them and what to pass. Two new rules were added because they came from real pain: no goodbye while a question is still open, and the “wait” tool only if the person asks to wait.

In the 30 inbound scenario test battery, Haiku with the long prompt met 79% of criteria. Haiku with the short prompt, second version, met 80%, never went silent, hung up before the goodbye 0 times in 60 runs, and had a median first word of 0.32 seconds. Luna on the same short prompt scored 74%, still rejected for Carla.

At 17:01 on 21 October, short Carla went live on the real line, on Haiku 4.5, with Gemini 3.6 chosen on purpose as fallback and the pre-generated “speculative” replies turned off. Same model that had failed five days before. Different prompt, different fallback, different result.

On 22 October I did short Andrea too. Prompt from 22,400 to 10,300 tokens, minus 54%, same model, same tools. On 55 real critical conversations, each replayed twice, short Andrea got 104 of 109 right versus 105 of 110 for the long prompt, and urgencies were 8 of 8. Shorter was not worse. On Carla it was slightly better.

3. Tools Must Say When Yes And When No

A tool description is not documentation, it is part of the behaviour of the agent. On 21 October the tool that tells the office “this person says they paid the bank transfer” fired on sentences like “I’ll do the transfer by Friday” and “What are these €300 for???”. Andrea told people who had not paid, “The office will check your transfer”, three times. Very polite. Very wrong.

The fix on 22 October was stupidly simple and absolutely necessary. The tool description now says when to use it and when not to, with near-miss sentences that must not trigger it: the future, the negation, the question. Same rule in the prompt, same rule in the test scenario. Positive and negative examples, not just “be careful”.

The same pattern showed up everywhere. The end-call tool got a new description saying “only after the goodbye, never after the person’s answer, question or choice”, and Luna’s early hang-ups on Carla went from 11–12 to 5 out of 30. Not enough to use Luna, but enough to prove the tool description is a lever.

Then there was Olevano. Three Italian towns share the name. On 17 October Andrea sent five houses near Rome to someone looking near Eboli. Now, when a name is ambiguous, the tool does not guess: it answers with a ready question listing each town in full, with province and region. There were 286 saved search profiles with a town name like that.

For tools that open work for the human team, there is also a server-side net. A plain check looks at the words passed in; if the sentence is clearly not the case, it opens nothing and tells the model what to say instead. When in doubt, it lets it through.

Every tool now has at least one test where it must fire and one where it must not fire, taken from real conversations that went wrong. All these rules live in one checklist for anyone who changes an agent. Every line was born from a real mistake, with its date and case.

4. Usage-Based Pricing Is A Nightmare

Create your cost report early, because “we will check it later” is how you wake up with the agents down for a whole night. On 6 October the monthly credit limit was hit and every conversation was refused. Not slowed down. Not degraded. Refused. With real people calling and writing.

The nasty part is that a lever can move only part of the bill. Andrea cost 43 credits per reply before the short prompt: 13 for the model, 30 as a fixed ElevenLabs platform fee for every written message. The short prompt can only cut the 13. After measuring 636 real replies, Andrea was at 39 credits per reply.

Carla had an even better trap. On her first evening live, short Carla on Haiku cost 14.5 model credits per second of call, more than Gemini 3.6 at 10.8. The cause was the “now” in the prompt: it read the platform system time, changing every minute. ElevenLabs caches the whole prompt, so every minute the whole block, about 29,700 tokens, was rewritten.

On Gemini this bug was invisible because cache writes are free. On Haiku a cache write costs 1.25× the full input price, while a cache read costs a tenth. Cache writes were 61% of the model spend, and 27 of 31 rewrites happened right after the minute changed. Beautiful little money fire.

The fix on 22 October was to freeze the time at the start of the call, like “Thursday 22 October 2026 at 08:53”. A test now fails if any prompt goes back to using a value that changes during the conversation. First full day after the fix, 68 calls, Carla used 8.0 model credits per second, one cache write per call. Cheaper than Gemini 3.6. And notice that the test battery could never have told me this: simulated runs do not report cache usage, and on 16 October the simulation rewrote the cache far less than the live line. Real cost is measured on real conversations.

Then there is fallback. ElevenLabs has a fallback model that kicks in when the main model is slow or fails, and many people do not know it is there. With the default fallback, on the calls of the afternoon of 21 October that were supposedly on Gemini 3.6, 34% of the model spend went to other models: Gemini 3.8, 2.5, GPT-4o. Andrea’s default fallback was GPT-4o.

With no fallback at all, you get “All LLMs have failed” and a lost call. With a random fallback, you get random cost and random behaviour. So choose it on purpose. Carla’s fallback is now Gemini 3.6, and it answered 2–3 turns in 12 calls, for $0.07.

Today I have a morning cost report with thresholds per vendor. ElevenLabs sells 11M credits a month for $1,100, and above that every credit costs 20% more. After the 22 October fixes, the agents run at about 8.9M credits a month, under the 11M included, against about 10.8M before. The tests of this billing cycle already took 1.3M credits, so every extra test battery this cycle is paid above the limit.

5. Prices And Models Move Under Your Feet

You never choose a model once, because prices, availability, and provider defaults keep changing while your agent is already live. On 1 October Carla and Andrea moved to Gemini Flash models: 3.6 for Carla, 3.8 for Andrea. The price was exactly half of 3.5, and it worked. Carla’s first word went from about 2.6 seconds to about 1.1 seconds.

That was the right choice then. But the half price is a list price valid only until 31 December 2026. From 2027 those Gemini models go back to full price, roughly 22 model credits per second on Carla, against 8.0 for short Carla on Haiku today.

And then the ground moved again. Google switches off Gemini 3.6 on 19 November. ElevenLabs started moving calls automatically to 3.8: 10% from about 20 October, 100% by 14 November. The provider was choosing the model for us, and that model would have broken calls.

Gemini 3.8 does not accept “minimal” reasoning. In tests it stayed silent in 18 of 30 scenarios with Carla’s settings, still 11 of 30 with a higher thinking budget, with a median first word of 5.3 seconds. So on 21 October the decision was: no more Gemini as candidates for Carla, Haiku instead.

6. Define The North Star Before You Drown

Define the North Star early, because otherwise you will optimise details and miss the KPI that actually matters to the customer. For this client, the real North Star is consultations booked per month: real people on a real call with a human consultant. Day to day, I read it as consultations booked per 100 contacts, so a busy week and a quiet week can be compared. After the 16 October change, people asking for a site visit are offered the phone consultation first. Over nine days, consultations booked per 100 contacts went from 25.8 to 31.4.

That is plus 22%, confirmed. This matters more than a cheaper reply or a faster first word. If Andrea uses fewer credits, good. If fewer people end up in a real consultation, who cares? Don’t fall in love with the technical metric just because it is easier to measure.

7. Expect Imperfection, Then Use It

Carla and Andrea are far from perfect, and the point is not to wait for perfection, because humans are not perfect either. I wrote about this before, and building these two agents made the point even stronger for me. Carla does not get tired at the end of the day. Andrea does not leave the night’s messages for the morning.

The work is to leverage the imperfection. You do not pretend the agent is always right. You design the flow so mistakes are caught, dangerous situations are escalated, and the agent knows when to stop. You also accept that some imperfections are cheaper and better than the current human alternative.

8. Humans In The Loop Are Still The Wildest Part

You can make Carla one second faster and Andrea cheaper by 76%, but you cannot prompt a human into opening the dashboard on time. Humans in the loop are still needed, and they are the least predictable part. In one week, 626 escalations were opened, and more than half only said “I want to talk to a person”. That is not a complex AI problem. That is an operational problem with a human expectation attached to it.

People kept waiting for a callback. Andrea had to be taught not to promise a call “as soon as possible”, because that sentence creates a debt someone else must pay. If the team calls after two hours, the user does not care that the agent was technically polite. The user remembers the promise.

And maybe this is the main lesson after building Carla and Andrea. The agent is not the product. The product is the whole machine around it: tests, costs, judges, dashboards, fallbacks, server nets, humans, and the uncomfortable discipline of checking again tomorrow what worked yesterday.

Because tomorrow, of course, the model will change.