The most valuable resources on innovation consistently suggest that innovation is a highly uncertain process. As a result, it is best to adopt an approach of experimentation and prototyping, continually iterating until you find the ideal product or market fit. This iterative process should be conducted with the same level of care and attention as the scientific method.

Despite this consistent advice, innovation metrics rarely include KPI measurements surrounding the specifics of experimentation and prototyping. Let's explore why this is so crucial and consider what a potential KPI might entail.

Fire bullets, then cannonballs

Great by Choice, a book by Jim Collins, introduces the concept of firing bullets before cannonballs. Innovations rarely happen in one big leap, but rather through a series of smaller steps that help to refine and achieve the desired outcome.

For instance, when Apple ventured into the retail industry, they began by opening a single store and testing various strategies to see what works best. Once they found the right approach, they expanded and eventually became the most profitable retailer per square foot globally.

So, what makes a bullet?

  • The cost of a bullet is low. However, the definition of low cost varies depending on the size of the enterprise. 
  • The risk of a bullet is low. However, that doesn’t mean a low probability of success, but it means minimal consequences if the bullet hits nothing.
  • The distraction caused by a bullet is low. It might be a high distraction for a few individuals working with it, but for the enterprise as a whole, it’s a very small interruption.

Disciplined Experimentation 

The book "The Other Side of Innovation" by Govindarajan and Trimble provides in-depth guidance on how to prepare yourself for innovation. The authors emphasize that generating ideas is not the most challenging aspect; execution is where you should concentrate your efforts. The crucial element of successful execution is the ability to conduct disciplined experiments.

In managing their ongoing operations, companies strive for performance discipline. For the innovation initiatives, however, they ought to strive for discipline of a different form: disciplined experimentation. Indeed, all innovation initiatives, regardless of size, duration, or purpose, are projects with uncertain outcomes. They are experiments.

To bring an idea to fruition, it must first undergo the transformative process of experimentation. Begin by crafting a detailed plan, complete with a scorecard that outlines the assumptions and data points to be measured. Develop a concise hypothesis, expressed in simple terms, that articulates your anticipated outcomes. Then, seek opportunities to invest a small amount of resources while gaining a wealth of knowledge. Continuously evaluate and adjust your plan as you progress, allowing for formal revisions to the initial predictions.

...the scientific method is the innovator’s indispensable friend … when running innovation initiatives, businesspeople need to behave more like scientists.

The 5x5 Methodology

According to Michael Schrage's book The Innovator's Hypothesis, the key factor that sets innovators apart from non-innovators is their commitment to experimentation.

Look carefully at the history of technology, entrepreneurship, or business innovation. A persistent pattern emerges. Successful innovators talk about ideas, but they invest their time, money, and ingenuity in expressive experimentation. Their competitive success comes from getting more value faster from expressive experimentation.

Ultimately, the goal is to gain valuable insights that bring you closer to achieving innovation. It's like purchasing a dollar's worth of innovation insight for only 50 cents, or even 20 cents. This approach allows for fast and cost-effective experimentation, resulting in a high return on investment through validated learning.

Give a diverse team of 5 people no more than 5 days to come up with a portfolio of 5 business experiments that cost no more than $5,000 (each) and take no longer than 5 weeks to run.

After the experiments are completed, the findings are presented to the senior management board, who then decides which ideas will receive further funding. The ultimate objective is to create a diverse portfolio of experiments. However, many organizations struggle with the process of designing and managing such a portfolio. This is where the 5x5 methodology comes in, providing a quick and effective solution to get started and overcome this challenge.

Schrage delves into the intricacies of implementing and evaluating the 5x5 method, showcasing captivating examples that make for an engaging read. This approach is not only easily achievable but also holds immense potential for fostering innovation and strengthening one's ability to innovate. But as he notes, big organizations find it incredibly hard to instill as a discipline:

Creating simple, compelling, and readily testable business hypotheses is managerially unfamiliar, uncomfortable, and unrewarding. So managers avoid them.

The DNA of an Innovator

In their groundbreaking book, The Innovator's DNA, esteemed business experts Jeff Dyer, Hal Gregersen, and Clayton Christensen delve deep into the traits of successful innovators. They identify five essential skills that aspiring disrupters must master: Associating, Questioning, Observing, Networking, and, perhaps most importantly, Experimenting. These skills form the DNA of an innovator, allowing individuals to push the boundaries of what is possible and drive forward groundbreaking ideas.

Their analysis revealed that innovators of all kinds, including not only startup entrepreneurs but also process innovators in large organizations, exhibited significantly higher skills in experimentation.

One of the many example innovators in the study is Jeff Bezos, who puts the ability to experiment at scale at the heart of Amazon's innovation strategy:

Bezos’s experience has taught him that experimenting is so critical to innovation that he has tried to institutionalize it at Amazon. “Experiments are key to innovation because they rarely turn out as you expect, and you learn so much … I encourage our employees to go down blind alleys and experiment. We’ve tried to reduce the cost of doing experiments so that we can do more of them. If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce.”

To learn more about the five skills and how to develop them, check this out.

Build, Measure, Learn

The release of The Lean Startup was a game-changer, perfectly capturing the essence and approach to innovation in the modern era. It sent large corporations into a frenzy, as they tried to decipher how to apply its principles to their own environments, resulting in a surge of lean startup consultants. The brilliance of its concepts lies in their simplicity and effectiveness.

Develop a hypothesis, establish a method to put it to the test and create a system to closely monitor the results. Then, after each rapid iteration through this process, make the decision to either persist and continue with a revised hypothesis and new implementation, or change direction and explore a different hypothesis. This approach is effective for making small incremental improvements as well as launching entire products.

The essential lesson is not that everyone should be shipping fifty times per day but that by reducing batch size, we can get through the Build-Measure-Learn feedback loop more quickly than our competitors can. The ability to learn faster from customers is the essential competitive advantage that startups must possess.

The Lean Startup works only if we are able to build an organization as adaptable and fast as the challenges it faces. This requires tackling the human challenges inherent in this new way of working.

How to measure experimentation

So if experimentation is so widely regarded as the basis for innovation, why do we rarely see KPIs in place to track it? After all, what gets measured, gets done, right?

If you’re using Idea Campaigns to generate high quality ideas, which target a known problem, with a sponsor who backs (and funds) selected ideas. Ideas are rarely ready for implementation right away, they need to be developed, or combined with other ideas, and then tested. This is where Concepts come in.

The two critical KPIs are then:

  • Number of experiments in my innovation portfolio (number of Concepts created)
  • Number of iterations through those experiments (cycles through the build, measure, learn cycle)

Seeing these numbers go up over time, gives you a direct pulse on whether you are building innovation muscle. Providing you are following the scientific method, and seeking out validated learning, not just experimenting with no rigor.

Supporting these metrics, you’ll also want to know more granular details about the portfolio, such as:

  • Number of experiments being run per "strategic innovation area," to show you the health of each major area you’re targeting.
  • Number of experiments by their status (pivot, persevere, or whatever other status you will determine for experiments).
  • Number of experiments per organizational unit / business division / department, to help you understand what areas of the business are "getting it," or struggling. Experimentation is not the exclusive domain of the innovation department, it should be part of the culture across the entire organization.
  • You may also want to measure the time to cycle through an iteration. As Eric Ries notes, the question is how fast can you get through each loop, so that you’re learning faster and moving closer to the right outcomes?

We’re still spending a lot of time thinking about number of ideas, number of participants, and other rudimentary KPIs. These are nice to know, but they don’t tell you anything about your progress towards a more innovation-capable organization. The back-end is where the true health of your program is measured, and for that, we must measure and promote the experiments. As Michael Schrage nicely puts it in The Innovator’s Hypothesis:

Good ideas have nothing to do with good implementations … Implementation - not the idea - is the superior unit of analysis for assessing value creation. How organizations enact ideas - not the ideas themselves - is the soul and substance of innovation. More often than not, implementation ends up redefining both the boundaries and the essence of the original idea.

Are you tracking how you experiment? I’d love to know.

Jump to Section