Why most AI pilots never reach production, and what we do differently

We inherited three different AI pilots from other vendors last year, each one built by a different team, each one stuck at the same stage: a working demo that had impressed a room of stakeholders, and then nothing, for months, while the actual production rollout quietly stalled. Looking at what those three pilots had in common taught us more about where AI projects actually die than any of our own successful launches did.

A demo needs to work on the ten to twenty examples someone hand-picked to show it off. A production system needs to work on whatever a real user throws at it on a Tuesday afternoon, including the inputs nobody thought to test. All three inherited pilots had been built and evaluated entirely against curated example sets, with no real evaluation against the messy range of actual production traffic, so the gap between demo performance and real performance only became visible once someone tried to actually ship it, at which point the accuracy numbers that had looked so promising in the pitch meeting fell apart.

We treat the demo and the production build as genuinely separate phases of work now, with a real evaluation set built from actual historical data standing between them, specifically because skipping that step is the single most common reason we see otherwise promising pilots stall.

Two of the three pilots had no defined fallback behavior at all. The assumption, unstated but clearly baked into the build, was that the model would simply be right, and nobody had actually designed for the case where it wasn't. That's a fine assumption for a demo where every input is chosen to succeed, and a serious liability the moment real users start sending genuinely ambiguous or out-of-scope requests the system was never built to handle gracefully.

Every system we ship now has an explicit answer to what happens on a low-confidence output before a single line of the happy path gets written: escalate to a human, return a defined fallback response, or decline to answer rather than guess. Designing that path first, rather than bolting it on after a launch goes wrong, is one of the clearest differences between a pilot that stalls and one that actually reaches production.

None of the three inherited pilots had a real cost projection attached to them, just a vague sense that API costs would be manageable. Once we ran the actual math against expected production volume for one of them, the projected monthly cost was close to four times what the client's finance team had assumed, entirely because nobody had modeled the difference between demo-scale usage and full production traffic, including retries, longer context windows as usage grew, and the cost of running an evaluation pipeline alongside the live system.

A finance team that gets surprised by a cost number after launch becomes a finance team that kills the project regardless of how well it actually works. We build the cost model before writing production code now, sized to realistic production volume, specifically so that conversation happens on our terms, early, rather than as a crisis three months after launch.

For all three inherited projects, the path forward looked less like new AI work and more like ordinary infrastructure work: building a real evaluation set from historical data, defining explicit fallback behavior for low-confidence cases, and producing an honest cost model sized to real volume. None of that is the exciting part of an AI project, and none of it shows up well in a demo. All of it is the actual difference between a system that gets quietly shelved after impressing a room once and a system that's still running, unremarkably and reliably, a year later. We'd rather spend the first month of any AI engagement on this unglamorous work than deliver an impressive demo that never makes it past the pilot stage.

In two of the three cases, the original vendor team had moved on to a different client entirely by the time we were brought in, leaving behind code with no real documentation of why specific decisions had been made: why this particular confidence threshold, why this particular prompt structure, why certain edge cases were handled the way they were. Reconstructing that reasoning took real time, time that could have gone toward actually fixing the gaps we'd identified, and it's a cost that rarely gets factored into an initial AI pilot's budget, since everyone assumes the same team will still be around when it's time to move from pilot to production.

We build every system now with the explicit assumption that a different team, possibly including the client's own future in-house engineers, will need to understand and modify it without us in the room. That means real documentation of every threshold and design decision, written at the time the decision gets made rather than reconstructed later from memory, and it means avoiding clever, undocumented shortcuts that make sense to the person who wrote them and to nobody else six months on. None of this is unique advice for AI systems specifically; it's just ordinary engineering discipline that got skipped in all three of the pilots we inherited, apparently because the fast-moving, exploratory feel of early AI work made it feel like documentation could wait until later. It couldn't, and by the time later arrived, the person who could have written it accurately was already gone.

We now treat a walkthrough session with whoever inherits the system, even if that's a hypothetical future hire the client hasn't made yet, as a deliverable in its own right, recorded and written up alongside the code. It's a small addition to the project timeline, and it's also why we now build in a deliberate month of shadow operation before fully retiring the process a new system replaces, specifically so a documented, working fallback exists if a subtle issue surfaces only after the original team's context has already left the building.