5 Bitter Lessons of AI Business Value

It’s becoming increasingly common to hear the following sentiment in large companies: “Our AI proofs-of-concept looked amazing, but it’s not showing up on the P&L.”  Most companies blame the use-case selection, but my experience says otherwise. Success is possible. Successful winners embraded the following bitter truths on what it really takes to get the value, whereas the disappointed majority was hoping tech alone will save them.

1. One use-case (in a domain) won’t move the needle

Most pilots automate a sliver of the work—drafting an email, summarizing a meeting—so the overall process still crawls at the pace of the next manual step.  Solve one bottleneck and you simply expose another.  The exception is scale domains where hundreds of people perform the same task (think contact-center agents answering simple requests). Everywhere else, the playbook is a portfolio of interlocking use-cases that together reshape the workflow in a domain (E.g., sales, marketing)

What to do: Budget for a portfolio, not a hero project. Commit to shipping at least three inter-locking use-cases in the first 12 months, all in the same domain and served by a shared data assets. Ask for investment approval based on ROI of the portfolio, but ask for investment as a VC - enough to develop MVP of the first use-case.

2. Rewrite the work process, not just the toolbelt

Treating an AI tool as a bolt-on preserves yesterday’s process and yesterday’s org chart.  Step-change value appears only when you redesign the sequence around what the AI systems do well and what humans need to do.  If your AI roll-out doesn’t have any meaningful re-thinking of the process, you may be heading for the land of micro-productivity. For example, in tech, give AI to writers speeds product-marketing copy only 5-10 % because the Product Manager (PM) is still the bottleneck; instead, arm the PM to draft the first 80 % and let writers polish, and the workflow cam accelerate by 40 %.

What to do: Redesign how the work gets done before you start the implementation. Usually this starts by understanding how the work gets done today and what assumptions (usually unspoken) about how the world worked made that process make sense. Focus deeply on how those assumptions no longer hold true and reorganize appropriately.

3. Your data is messy, but you don’t have to boil the ocean

LLMs leverages whatever the retrieval layer feeds their context.  Unfortunately, the data that truly matters such as rationale for pricing exceptions, troubleshooting logs, customer conversations are often scattered or not collected at all. Even when the data is collected and in one place it’s often full or low quality mixed in with high quality (e.g., a bunch of mediocre sales presentations mixed in with a few winning ones). The fix is incremental: pick the narrow slice of data that powers the first use-case, clean that to production grade, and expand from there. Often the data for 2nd or 3rd use-case isn’t even collected today so this gives you time to start collecting it.

What to do: Plan for ongoing data prep and SMEs to support it, but continuous enrichment beats heroic one-off clean-ups. Keep teams small who have the right expertise and track high-quality documents added per sprint.

4. Treat your AI solution, like a living tech product

No two procurement, or sales processes are twins. An AI assistant or agent dropped into a process must be configured, tuned and iterated to match the actual process. Without it you are usually implementing tools for some platonic ideal of the work process (usually from vendor’s perspective). This introduces lots of friction because humans must act as glue between real process and imagined process. Even within a company, different teams have different data sources, use different process, must follow different regulations, etc. This is why you should scale your AI solution slowly to allow you to configure them properly.

What to do: Appoint a Product Manager (not Project Manger) and a cast of supporters (Data SMEs, data engineers, etc.) that guide how all the functionality is built out with frequent input from the actual end users for a year or even two. Good PMs are rare, and AI PMs are even rarer, but they can make a huge difference in creating a set of tools that delight vs those that become AI shelfware.

5. Pair carrot with stick for adoption

Humans often optimize for comfort and status quo so some incentives are necessary to break through. This is especially true as over the last 10-15 years constant parade of SaaS tools have promised breakthroughs but often delivered just more process, resulting in apathy towards new tools. Most companies opt for a carrot approach, a few for the stick but the best do both. The carrot is creating a tool that genuinely makes the day shorter or the task easier. The most successful AI programs are not AI programs but “Improve <function X> experience”. The stick is a clear top-down mandate: goals with metrics and often clarity on how freed-up time or money is reinvested. It’s best if AI-enabled targets are baked into operating goals.

What to do: If the AI portfolio you are trying to adopt isn’t a top 3 priority for the executives, then it’s likely dead in the water because you won’t have the stick. Similarly, if your initial users aren’t huge advocates, you don’t have product-market-fit necessary.

Conclusion

These lessons feel bitter because the work to overcome them is hard compared to how easy it is to get tech to work (at least in pilot).  Most firms nibble around the edges and then lament the cost of AI tech. The few that swallow the pills and the work that comes with them bank the upside. It is precisely because this requires a lot of work that companies should be very thoughtful around a few bets where they can afford the tech and people investment that will have a high ROI.

1  Yes the article is a nod to the far more famous “Bitter lesson” of AI by Rich Sutton