The fast-start GenAI trap

There's a growing sense of disappointment (e.g., here and here) around the potential of AI in the Enterprise. Many point the finger at the technology, but I believe the issue is more often with the choice of application and how it is developed and deployed. The previous post discussed how the choice of use cases and how to deploy them could impact value capture. In this post, the focus is on one under-discussed element of product management when companies build their own AI use cases. 

Many organizations see promising results during the initial proof of concept (POC) stage, where the goal is to prove that the system can perform the needed task(s) but not yet that it can create business value. Bain’s survey from Q1 showed that around 75% of GenAI use cases met or exceeded expectations, and many of those use cases were in the POC stage.

The real challenge begins when moving from a successful POC to a minimum viable product (MVP) that needs to work in real-world conditions and demonstrate tangible business value. For Gen AI use cases, time to POC tends to be shorter whereas time to MVP is often longer. This increasing delta between time to a POC and time to a MVP is a key—and often overlooked—reason for the frustration many business leaders face with AI initiatives.

First, let’s examine what creates this delta and then observe how it can result in early failure for an AI project.

It is easy to create a GenAI POC quickly for several reasons.

• Capable pre-trained models: AI models today are pre-trained to perform many different tasks, have rudimentary reasoning and can even handle image inputs. With Traditional AI/ML, one would need to train a new model for every task which could take months.
  

• Viable with little data: POCs often require minimal data to show the art of the possible. This is because modern models can perform many tasks out of the box and need just a few examples to “learn” how to perform a task, even as they are “asked” to perform it. The fancy term for this is “in-context learning”.
  

• Pre-built templates: Lastly, a POC can be quickly assembled using tools, templates, and frameworks from start-ups (e.g., LangChain, LlamaIndex), model providers, or cloud providers for most common use cases like chatbots or knowledge management assistants.

However, moving from POC to MVP is a completely different story. The technical hurdles to reach a production-ready system appear more complex:

• Invisible data quality issues—Many GenAI systems depend on unstructured data from operational systems that are not well governed (e.g., support tickets, SharePoints, and crowd-sourced knowledge articles). This ungoverned data can suffer from issues like duplicates, conflicting articles, incomplete articles, and out-of-date information. Building GenAI systems usually exposes those data quality issues that have existed all along but were invisible.
  

• New tech platform - AI systems that leverage GenAI require some new tools (e.g., vector databases, AI gateways, guardrails, prompt-injection checkers) and many companies don’t have them on their platforms. Worse, for some components, tools don’t exist or are immature, which results in the need for custom work.

• New patterns of building - Furthermore, these systems are new, and best practices are not yet well known. These systems fail in ways that are different from traditional software. We have decades of experience with traditional software, but few product managers and technical leaders have enough experience to predict and avoid traps and bottlenecks with AI systems. The challenge is compounded by the fact that AI systems can be non-deterministic and talent with real experience is scarce.

The aftermath:

Unfortunately, business and tech leaders often misinterpret POC success as a signal that GenAI development is simply faster. There are 2 critical traps here:

1. Shorten time to MVP: Leaders extrapolate speed to POC and thus assume that the typical time to MVP will be shortened by the same amount.

2. Set ambitious scope: Leaders sign up for MVP scope that handles the entire domain space. For example, an ambitious scope is a customer service bot that can handle all customer support queries. A more realistic scope could be to handle a fraction of queries where data is good and connections to other systems are already available.

The result of these two traps is that projects fall behind schedule or can’t handle all the required tasks with sufficient accuracy. When this happens, it’s not rare for executives to lose confidence abandon the project, and declare AI to be overhyped. 

The lesson for leaders is clear: the road from POC to MVP can be long, and it’s essential to plan accordingly. Success at the POC stage is only the beginning. Managing the scope of an MVP is crucial, and organizations should focus on domains where the data is clean and can deliver immediate value to a defined group. It’s OK to trade off a smaller scope (e.g., fewer features or relevant to a smaller group) to gain higher accuracy. As the data is improved, the AI system can be expanded to broader use cases. 

Conclusion:
The fast success with GenAI POCs can lead to premature optimism. This optimism results in overly ambitious scope and timelines for MVPs, which are typically more difficult to achieve than anticipated. The most successful leaders will paint an ambitious roadmap but avoid overconfidence from a fast start. They will manage timeline expectations and ensure a realistic scope for the first step - the build of the MVP. By doing so they will avoid one trap of AI disappointment and build a successful MVP that will give them momentum and a foundation for long-term success.