Six months in: what we've learned shipping AI agents for clients

We shipped our first fully agentic system into production a little over six months ago, and we've since launched four more across clients in logistics, insurance, and professional services. Looking back at that first batch honestly, several of our early assumptions about what would matter most turned out to be wrong, and the things that actually determined success weren't the ones we spent the most time worrying about going in.

Our first agentic project involved weeks of comparative testing across different underlying models before we picked one, chasing marginal accuracy differences that, in retrospect, mattered far less to the system's real-world performance than the quality of the tool descriptions and guardrails we gave it. A well-specified toolset with clear boundaries on what the agent could and couldn't do consistently produced better, more predictable behavior than a marginally more capable model with a loosely specified toolset. We spend a lot less time on model selection now and a lot more time on tool design, and the systems have gotten more reliable as a result, not less.

Across all five projects, defining exactly what the agent was and wasn't allowed to do, and building the enforcement for those boundaries, consistently took longer than building the agent's core reasoning loop. That surprised us on the first project and stopped surprising us by the third. It makes sense in hindsight: the core loop is a well-understood pattern with plenty of existing tooling to lean on, while the specific boundaries of what's safe for a given client's specific operational context have to be worked out from scratch every time, in detail, with the client's own team. We now scope guardrail definition as its own explicit phase of every agentic project, with its own timeline, rather than folding it into general development time where it tends to get compressed under deadline pressure.

We'd braced for a slow trust-building period where clients would double-check the agent's output closely before relaxing into normal usage. In practice, four of the five clients started trusting the system's output within the first two to three weeks, faster than we'd planned for, which shifted the risk profile in a direction we hadn't fully anticipated: the danger wasn't distrust slowing adoption, it was trust outpacing the system's actual track record. We've since added a deliberate, gradual expansion of the agent's autonomy over the first month of any new deployment, specifically to make sure trust and actual demonstrated reliability grow at the same pace, rather than letting early good performance justify skipping review steps sooner than the evidence actually supports.

Based on these five projects, every new agentic engagement now starts with two weeks explicitly scoped for tool and guardrail design before any reasoning-loop code gets written, uses a fixed, deliberate ramp for expanding the agent's autonomy over its first month in production regardless of how well early results look, and treats model selection as a comparatively minor decision to revisit later rather than a gating decision made upfront. None of these changes came from reading about best practices; they came from watching where our own first batch of production agents actually struggled, which is exactly the kind of lesson that six months of real client deployments teaches faster than any amount of research would have.

The professional services engagement in that first batch of five is the one we'd restructure most if we were starting it again now. We built the agent's guardrails around a set of assumptions about how staff would actually use it that turned out to be wrong within the first month: we'd assumed most interactions would be short, single-step requests, and had scoped the review process accordingly, but staff quickly started chaining several requests together in ways that meant the agent was accumulating context across a session in a way our original guardrail design hadn't fully accounted for. Nothing broke in a dangerous way, but the review process we'd built was reviewing individual actions in isolation when the more useful thing to review, in hindsight, was the full session as a coherent sequence.

Rebuilding that review process mid-engagement cost us roughly three weeks we hadn't budgeted for, and it's directly shaped how we scope every project since: we now explicitly ask, before any guardrail work begins, whether real usage is likely to be single-step or session-based, and we design the review and logging infrastructure around whichever answer the client's actual workflow suggests, rather than defaulting to the simpler single-step assumption because it's easier to build first. It's a small question with a large downstream effect on how much rework a project needs once real usage patterns diverge from what was originally assumed.

We also underestimated, on that same project, how much staff would want to understand why the agent had made a particular decision, not just what decision it had made. Our original interface surfaced the agent's final action clearly but not the reasoning behind it, and staff quickly started asking us directly, rather than trusting the system, whenever an action looked even slightly unexpected. Adding a short, plain-language explanation alongside every proposed action, not a full technical trace, just one or two sentences on why this particular step was chosen, cut those direct questions to us by more than half within the first two weeks of that change shipping. We now build that explanation layer into every agentic system from the start rather than treating it as an optional nicety to add later if staff ask for it, since staff reliably do ask, and building it retroactively costs more than building it in from day one. Between the guardrail rework and the added explanation layer, this single project taught us more about what actually matters in production agentic work than the other four combined, if only because it was the one where our original assumptions were most visibly wrong in ways we couldn't ignore once real staff started using the system daily.