A large part of a testing consultant’s job is wrapped up in educating clients about what it means for an organization to be optimization-mature. That’s because optimization affects all parts of an organization.

In optimization-mature businesses, people understand why experimentation matters. They vet new ideas through a consistent, repeatable process of evidence-based research. And they put their egos to the side in the service of what’s best for the business.

None of these come naturally to a business – or to business owners. That’s why it matters, at least for the time being, for those in charge of optimization to act in a consultative manner, working directly with executives to plan strategy, marshal buy-in, and get things done.

Resources for understanding optimization maturity

In 2018, WiderFunnel published an Optimizely-funded research report on the current state of optimization maturity across organizations of all sizes below the Fortune 50. At the very top lie the winners in the tech economy – Amazon, eBay, Netflix, et. al. – where optimization drives all components of the business.

Tellingly, WiderFunnel’s report also mentions that almost two-thirds of surveyed businesses have clear executive buy-in for optimization.

In the past, I’ve also cited the CRO Maturity Model, proposed by Michal Parizek to indicate how prized optimization is within the organization. (Here’s a full-sized version of the model; the aforelinked site’s full-size image is a 404):

Fresh Egg also has a four-step CRO model – although the first step is “doesn’t optimize”, which is a bit peculiar. You probably don’t want to be at that stage for long!

What unites these models

What matters between each of these models? Mindset.

First comes awareness.

As in Dunning-Kruger, you can’t focus on optimization if you don’t know that it exists.

But it’s one thing to be aware of optimization, another thing to understand that it has value, and then another thing entirely to understand that it has value and can be applied to your business.

Then comes process.

Hilariously, numbers don’t matter when it comes to optimization maturity. “Experiments run per month” is a poor proxy for optimization maturity. So is the amount of money you pour into optimization. How you deal with new design decisions is what really matters.

Design affects the fundamental politics of any organization. It’s the fundamental shape of how you communicate with customers, how you treat them when they come in the door, and how you handle when things go wrong. Those who control the organization’s design controls the destiny of the organization.

Optimization-mature organizations need a clear, articulated process for discussing, researching, prototyping, testing, and launching new design decisions.

Finally comes instinct.

There should be no debate on what to do when contemplating a new design decision. Better to resort to the process that’s already been built – and to recognize that it’s put in place for a reason.

This applies at every level of the organization, right up to the CEO and founders.

Optimization-mature businesses run lots of tests not because they’re mature, but as a result of it. They have a higher share of winners, too, because they’re mature enough to throw away decisions that are unlikely to move the needle for the business.

Characteristics of optimization-mature organizations

Once you’ve gone through this process, what does optimization maturity really look like?

• More server-side tests. The less you rely on your WYSIWYG testing framework and start putting together more split-page tests, the more mature you are as an organization. If you roll your own in-house framework, and start creating tests that only fire server-side, then you’re likely devoting significant programming resources to running tests – and doubling down on a long-term optimization strategy.
• A consistent routine of research. Research doesn’t happen once, and it doesn’t happen in a vacuum. Optimization-mature organizations are researching constantly, and always questioning their old ideas.
• Solid procedures for managing new research. New ideas for design changes are great, but researched ideas win. Optimization-mature organizations understand this, and put all new ideas into a research process – in order to validate their unfocused hunches, and shape them into an improved design.
• Executive buy-in. Optimization fares better with executive buy-in than without. You get the right resources, you get around HiPPO problems, and you get a cultural shift that favors optimization.

What are the consequences of optimization maturity? You know you’re doing things right when you get:

• A win rate over 50%. Since researched tests win more often, and since the team gets more informed about what tests win over time, it’s likely that the overall win rate is likely to increase over time, as you make fewer mistakes.
• Consistent reuse of test results. Test results are a form of research. Optimization-mature organizations know not to retread ground that isn’t working out for them.
• Consistent maybe or no responses to executive requests. Nothing happens on a whim in optimization-mature organizations – nor is it reactive to competitors for its own sake. Businesses know to respond deliberately and intentionally when they understand the value of research-driven testing.

← Back to the Blog Check Out Draft’s Store →