<img height="1" width="1" src="https://www.facebook.com/tr?id=1483239291704574&amp;ev=PageView&amp;noscript=1">

The value of experiments for innovation is a topic we’ve previously covered here on the blog, and for good reason: if done well, experiments can help avoid some costly new product, service, process, and even business model mistakes. We’ve also written extensively about change, because managing innovation is nothing more than managing transitions and transformations of all parties involved: customers, suppliers, internal stakeholders and even competition. As Microsoft likes to put it “[our] IT stands for Innovation and Transformation”.

What about bringing the two together? Can organizations effectively experiment with change? Thanks to powerful software, as well as collaboration with universities & research institutes, they can. In fact, today innovation leaders have more options than ever to create simulations and design small scale tests before making the next bold move.

swedish_traffic_switches_sides.jpgMajor change: Swedish traffic switches sides on September 3rd, 1967 (source)

In as follows, some essential points to consider when thinking about your next business experiment and the must-reads that contain them:

Experimentation is ubiquitous but not all of it is necessary or good

In his book, Experimentation Matters (2003), Stefan Thomke writes that tasks like investigating what reward schemes lead to higher productivity, or which software tweak will finally remove that persistent error are “every day” experiments employees do naturally. From this standpoint, experimentation is ubiquitous but not all of it is useful or “sound”. Good experimentation reaches well beyond the individual and is used to manage various sources of uncertainty in the innovation process. 

“It is important to note that good experimentation goes well beyond the individual tests and their protocols. It is about the way in which companies use such tests to manage, organize, and structure their innovation processes”

In Why Companies Should Have Open Business Models (2007) Henry Chesbrough makes a similar point and provides an example: in its attempt to create healthier, shelf-stable products to its customers, a food manufacturer uses a “white box” brand that is not advertised, unsupported and with no obvious connection to the company. This provides the ideal “test bed” as companies are generally hesitant to launch experiments that might risk the reputation of an established brand. Through this approach, experiments become very powerful indicators of success at a fraction of the risk normally involved.

Underestimating an experiment’s worth

Managers underestimate the worth of experimenting (with change) for a variety of reasons, most of them falling into the “cognitive biases” category (Dan Ariely’s podcast, Arming the Donkeys, is an excellent resource here by the way): confirmation bias, availability heuristics (1), escalation of commitment, framing effects (2), the popular halo effect (3) and, my personal favourite, the “history-of-ownership effect” whereby people who owned an object for a long time believe it is worth more than the people who have just acquired the object.

From that list, confirmation bias and escalation of commitment in particular, according to Beshears and Gino, systematically lead firms to refrain from evaluating changes because the key decision makers feel (erroneously) that they already know that the changes are good ones. A select few, however, see the opportunity of challenging their instincts and running proper tests. Pressing questions can lead to interesting experiments and learning about the relationships between cause and effect can offer valuable paths to development.


Source: Mark Anderson

And the costs of not trying

Generally speaking, the costs of not evaluating changes can be high and often hidden. Take, for example, the introduction of a client relationship management (CRM) system: implementing it leads to a 15% increase in revenue but perhaps not implementing it leads to a 20% increase. This is the hidden cost of not experimenting according to the same Beshears and Gino. Ben & Jerry’s CEO, Jostein Solheim, knows exactly what this means. In a recent intervention at the Client Conference in Paris he and his team showed what ice cream looks like if it’s just two degrees warmer: “It’s a bit of a mess”. His mission therefore became to help climate change negotiations chill-out one scoop at a time – a powerful experiment and metaphor!

On a similar note, Michael Schrage writes that when experimenting with change - or with anything else for that matter, it’s also important to keep in mind common sense rules. In The Innovator’s Hypothesis, he points to the following:

  • clarify the target outcome (because narrowly defined outcomes will not help!);
  • articulate well what the proposed changes will involve - the simpler, the better;
  • introduce changes in some places of the organization but not in others - they should bear some common characteristics, of course – and use “treatment groups” and “control groups”.

Fast, frugal and fruitful experiments using the 5x5x5 design

When it comes to designing and implementing so-called fast, frugal and ultimately fruitful experiments, the same Michael Schrage explains in Serious Play (1999), that blue sky/ anything goes experiments are not the answer. Instead, friendly rivalry between teams, sharp deadlines and experimenting in a way that commands strategic attention and respect, are healthy substitutes. To this end, he points to the 5x5x5 model in which a minimum of 5 teams of 5 people are given no more than 5 days to come up with a portfolio of 5 business experiments that should take no longer than 5 weeks to run and cost no more than 5,000 EUR to conduct. Each experiment is attached to a business case, explaining how running that experiment gives insight into a possible 5 million EUR savings or 5 million EUR growth opportunity for the firm. In his article for the European Financial Review, Schrage exemplifies how this model might be used to create experiments in the financial services sector (debit and credit card swipe machines), or in the IT service sector (call centres). 

Altering the blueprints: the most challenging experiment of all  

Altering an organization’s “blueprint” (4) – or its internal process - is intuitively disruptive and destabilizing and likely the most difficult experiment organizations might ever undertake. After all, abruptly changing the way people are rewarded for their efforts or in the way they report to top management can have dramatic effects on their performance.


Source: Baron, Hannan, and Burton (2001)

In an analysis of high-technology start-ups in Silicon Valley, professors Baron, Hannan and Burton found that changes in the employment models or blueprints embraced by organizational leaders increase turnover, but adversely affect organizational performance. The magnitude of the effect of change depends on the “destination” – or which new blueprint (employment model) they want to migrate to. For example, the movement to bureaucracy from either the engineering model or an aberrant blueprint produces significantly less turnover than the abandonment of another basic model (especially the star type) for bureaucracy. Knowing his, innovation managers can design smooth transitions without risking a high attrition rate.

Summing up, whether a routine practice or a periodic “check”, experimentation is part of a healthy innovation mentality and should be pursued accordingly. As more organizations experiment with change, the better they will be quipped to brace whatever the future brings them.


(1) Judging probabilities of future events based on how easy those events are to retrieve from memory

(2) Reacting to a particular choice in different ways depending on whether it is presented as a loss or as a gain – e.g., a new product development could be framed as a chance to change the world, or as a change to embarrass the entire organization

(3) Assuming that because people are good at doing A they will be good at doing B, C and D

(4) For example, the star model refers to attachment based on challenging work, reliance on autonomy and professional control, and selecting elite personnel based on long-term potential, while the bureaucracy model involves attachment based on challenging work and/or opportunities for development, selecting individuals based on their qualifications for a particular role, and formalized control.

Related posts...

Jump to Section
Oana-Maria Pop

Oana-Maria Pop

Oana is the Head of Open Innovation at HYPE and specializes in the science of durable intra- and inter-firm relationships/collaboration. In her work, she combines academic insights with industry exper