Generative AI implementation: from pilot to production

The most common mistake in AI implementation is starting with the technology. A team decides to use an LLM, then looks for a problem to apply it to. The result is solutions searching for problems — technically impressive systems that do not address anything that actually matters to the organization.

Discovery works in the opposite direction. Start with the highest-friction workflows in your organization: the tasks that consume the most time, produce the most inconsistency, or create the most downstream problems when they go wrong. Map those workflows in detail — who does them, how often, what inputs they require, what outputs they produce, where things break down. Understand the current state before proposing a future one.

The output of discovery is not a technology plan. It is a precise problem definition: a description of the current workflow, why it is a problem, what a better version would look like, and how you would know if you had achieved it. Technology choices come after, and they follow from the problem definition rather than preceding it.

Time required: One to three weeks, depending on workflow complexity and stakeholder availability. Discovery that is rushed produces problem definitions that are too vague to guide build decisions. Discovery that expands indefinitely produces analysis paralysis. Set a time boundary and hold to it.

A pilot is a structured experiment. Its scope should be narrow — one use case, one team, one workflow segment — and its duration should be short, typically four to eight weeks. The narrowness is not a limitation; it is what makes the evidence legible. A pilot that tries to test everything tests nothing clearly.

The most important decision in pilot design is who the users are. The instinct is to start with the most enthusiastic adopters — the people who volunteered for the pilot, who are excited about AI, who will give generous feedback. This is exactly wrong. Enthusiastic early adopters will use a mediocre system and report positive results because they are motivated to. The users you need in a pilot are the most representative ones: people who will use the system exactly as it will be used in production, on real work, under real time pressure, and who will stop using it the moment it creates more friction than it removes.

Representative users surface the edge cases, the failure modes, and the workflow friction that enthusiastic users paper over. Their feedback is harder to hear and vastly more useful.

The go/no-go decision is the most skipped phase in AI implementation. Organizations that have invested effort in a pilot face organizational pressure to declare it a success — the team wants to continue the work, stakeholders want to show progress, and nobody wants to be the person who says it did not work.

This is exactly when honest evaluation is most important. Scaling a system that did not pass its pilot criteria is not optimism — it is the commitment of significantly more resources to a system with known performance problems. Those problems do not resolve themselves at scale; they amplify.

The go/no-go decision should be made against the success criteria defined before the pilot, by people with the authority to make a genuine decision rather than a formality. The output is one of three: go (scale as designed), redesign (the approach has merit but specific elements need to change before scaling), or stop (the evidence does not support further investment in this use case at this time).

The redesign outcome is the most common and the most underused. Treating a pilot as binary — full success or failure — misses the more useful finding: that the approach works in some conditions but not others, or that one component is strong while another needs redesign. A structured redesign decision preserves what works and fixes what does not, rather than either discarding the whole thing or scaling the problems.

Production hardening is the work that most technical articles about AI implementation skip, and the work that most implementations underestimate. A system that works reliably in a pilot — with an engineer available to handle edge cases, with known inputs, with manual monitoring — needs significant additional work before it is ready to run unsupervised at scale.

Guardrails and output validation. Production systems need explicit constraints on what the model can and cannot do, and validation logic that catches problematic outputs before they reach users. This includes input filtering (what prompts or content types should the system decline to process?), output validation (does the response meet format, length, and content requirements?), and fallback behavior (what happens when the model is uncertain, the input is out of distribution, or the response fails validation?).

Monitoring and observability. You cannot manage what you cannot see. A production AI system needs logging of inputs and outputs (subject to privacy requirements), alerting when performance degrades, and dashboards that make usage patterns and failure rates visible to the team responsible for the system. "We'll add monitoring later" is a sentence that predicts problems.

Cost management. LLM API costs scale linearly with usage. A system that costs $50/month in a pilot with 10 users may cost $5,000/month at 1,000 users — or more, if production inputs are longer or more complex than pilot inputs. Before scaling, calculate the cost at projected volume, verify it is within budget, and implement any necessary optimizations: prompt compression, caching of common responses, model tier selection based on task complexity.

Human override mechanisms. Every production AI system should have a clear and easy path for a human to intervene, correct, or override the system's output. This is not a sign of distrust in the technology — it is the accountability structure that makes it safe to deploy at scale. Users who cannot override an AI system will either stop using it or start ignoring the signals that tell them it is wrong.

A technically sound AI system that nobody uses has the same business value as a system that was never built. Organizational rollout — the change management, training, and workflow integration work that turns a production-ready system into a system people actually rely on — is the most commonly underbudgeted phase of AI implementation, and the one most responsible for the gap between investment and return.

Redesign the workflow, don't bolt the AI onto it. The instinct is to add AI as a step in an existing workflow: the person does what they always did, plus they now also use the AI tool. This rarely works. The workflows that generate the most value from AI are ones that have been genuinely redesigned around the AI's capabilities — where the AI handles the parts it is good at, humans handle the parts humans are better at, and the handoffs between them are explicit and well-designed. Bolting AI onto an unchanged workflow usually produces a workflow that is slower than before, because users are now doing everything they used to do plus managing an AI system on top of it.

Train for judgment, not just operation. Users need to understand not just how to use the system but when to trust it, when to be skeptical, and what to do when it is wrong. This is a higher bar than feature training. A user who knows how to operate the system but cannot assess the quality of its outputs is not equipped to use it safely. The failure mode to avoid: users who trust AI output uncritically because they were trained on what the system does rather than what it cannot do.

Name a workflow owner, not just a technical owner. Every AI system in production needs two owners: a technical owner responsible for the system's performance and reliability, and a workflow owner responsible for how the system is used, what training is provided, and how outcomes are measured. In many organizations, both roles end up with the same person — which is fine for small deployments but creates a single point of failure as the system scales.

A gen AI system that is not actively maintained will degrade. This is not a figure of speech. Base models are updated or deprecated. Usage patterns shift. The distribution of real-world inputs drifts from what the system was designed for. Regulatory requirements evolve. New capabilities become available that could significantly improve performance. Treating a launch as an endpoint rather than a beginning is the implementation equivalent of the pilot trap — a false sense of completion that sets up a slow, invisible failure.

Model drift is real and non-obvious. When an underlying model is updated by its provider, the behavior of your application can change without any change to your code. Outputs that previously met your quality bar may no longer do so. Monitoring catches this; the absence of monitoring means you find out from users — or, worse, from downstream consequences. Pin model versions where possible, and build a testing protocol for evaluating new versions before upgrading.

User feedback is a maintenance input, not a post-launch nicety. The most valuable signal for improving an AI system in production is structured feedback from the people using it daily. Build feedback mechanisms directly into the workflow — simple, low-friction ratings or flagging of poor outputs — and establish a regular cadence of reviewing that feedback and acting on it. Systems that collect feedback but do not act on it quickly lose user trust in the feedback mechanism itself.

The improvement backlog is never empty. Launch is not done; it is the point at which you have enough real-world data to meaningfully improve the system. Treat the backlog of prompt refinements, edge case handling, and workflow improvements as ongoing maintenance work with dedicated capacity, not as a list of items to address "when there's time."