Out now: Value-Based Design, the definitive way to prove your design’s worth. Read it.

How to Shift a Team Away from a “Move Fast & Break Things” Mindset

 

The most common mindset in any technically-oriented organization comes from Facebook’s now-abandoned motto: move fast and break things.

This mindset manifests in many different ways:

  • Changes made quickly based on suggestions from on high.
  • Deployment of bugs without proper QA.
  • Deployment of unresearched design decisions.
  • Wholesale abandonment of research.
  • Building without getting designers involved.
  • Frequently-shifting roadmaps.
  • Top-loaded meetings – or no meetings at all.

Setting aside the culture issues that this mindset brings about, this is not a good environment for those who want to optimize for conversion.

The organization is likely actively losing money on this process – not only because of the design-hostile, toxic culture that surely results, but also because they’re simply not coming up with reliably revenue-generating design decisions.

In practice, this results in the sort of A/B testing process where people stab at headlines and button colors, and then wonder why they aren’t moving the needle. Testing by CEO hunch doesn’t work. Testing by research does.

And research is antithetical to a “move fast & break things” mindset. It involves time, patience, humility, and rationality – all things that get thrown out the window when making rash decisions.

It is very easy to be hired to run A/B tests and then run into this. It causes significant friction, and if you don’t tackle it head-on, it’s going to make things miserable for you.

Why? A/B tests are a tool of optimization – and you’re not being hired to run tests, you’re being hired to increase conversions and AOV. But people view that through the idea of A/B tests. It’s on you to dial this back and convey what A/B testing really involves: a holistic, research-driven practice of optimization that requires rational, careful, intentional vetting of new design decisions.

Optimization is fundamentally a contrarian practice. It runs against the established wisdom in the industry – and as a result, it wins.

Let’s talk about the most common issues that appear in companies that haven’t fully reached conversion maturity, and how to tackle them one by one.

Rolling out unresearched changes

The whole point of optimization is to ensure that you’re making the right changes to the site’s design. That fundamentally requires research – before every change.

If someone is used to deploying lots of changes to the store without researching them first, that’s an opportunity to assert yourself as the point person on research and the gatekeeper of new changes. Roll back the change, explain the process, and come up with a way to research the person’s idea. Don’t do this on Slack; get them on a call to break what they’ll probably view as bad news.

During this, you need to draw a clear line between changes that need to be researched & tested, and one-off changes. Bugfixes or reductions in load time are usually safe to roll out to everyone. Everything else needs to be researched and tested.

Diminishing or eliminating research

Draft contractually enforces research with all of its clients. If a client says that research isn’t important or necessary, we have grounds to terminate with no refund.

If you’re an independent consultancy that performs research-driven optimization for many clients at once, you should do the same for all of your future contracts.

If you’re in-house, and upper management incorrectly deep-sixes research before making design decisions, it’s time to go guerrilla. You should do what you can to perform research on limited time, budget, and no administrative support.

If you want to read more on how to do it, my favorite book for these activities is Undercover User Experience Design by Cennydd Bowles & James Box.

Deploying multiple changes at once

Obviously, you can’t deploy two significant changes at once, because it’s impossible to tell if one or both helped or hurt the business.

This is a good opportunity to educate the team about multivariate testing. Instead of rolling out both changes at once, you can test both changes simultaneously – if you have enough traffic to do so.

Testing unresearched changes

Every single Draft Revise client has asked to test unresearched changes at least once. All of them. And it almost always comes from the CEO.

We usually solve this with our Trello board. Our kickoff call begins with an explanation of the board’s columns, and an entreaty to post all new test ideas to the “suggestions” column.

On the board itself, we have the following card:


This is where new test suggestions should be posted.

They can be high-level or grittily tactical. Once we’re ready to discuss, someone else (not you!) moves it into “Considering for Later” and asks questions.

Change suggestions should include:

  • The suggestion itself
  • What page it’ll occur on
  • What empirical evidence motivated it (e.g. customer service inquiry, GA findings, heat map, etc – this is required!)
  • Who’s suggesting it

Some important things to note with suggesting changes:

  • It does not matter who suggests what. The marketing intern should have as much weight as the CEO. In fact, the higher up people are, the more they should probably just listen.
  • It’s essential for us to get to the motivations behind design decisions. Let’s say you want to swap the headline & subheader. This is a cool idea! We may all even agree with it. But even so, we should take the time and energy to question what motivated us to suggest that, so we can figure out the best tactics for executing it. It could be that a swap and rewrite is better, or doing away with the subheader entirely.
  • It is okay if your suggestion is politely declined. We will be awash in suggestions, and nobody is trying to compete to have the best or coolest suggestion. More importantly, we’re all working towards a common goal: a revenue-generating design that helps grow the business.

One team member should be tasked with moving various changes across the columns as we create a consistent pipeline of tests across our whole funnel. When tests finish, they’ll end up in one of the three columns that says if a test won, lost, or was inconclusive. And if we decide to not go with a given idea, we’ll move it to the last column.


All test ideas need to be put through a consistent, repeatable research process before going live. That gives us greater insight into not only what to test, but how to test it.

Every time a test idea gets researched, it comes out improved – and it’s more likely to win, too.

Making changes to an active test page

This is an obvious no-no.

Before our first-ever round of tests, I provide a stern warning on Slack to not change the test page – and explain why that’s important. I also explain where to find what tests are currently active on Trello. Then I pin the message to our Slack room, so it can remain as prominent as possible.

Peeking

Tests need at least a week to run, often longer – and peeking can severely harm results. You know this, but other team members don’t.

The #1 thing you can do to reduce peeking is to severely restrict the number of people who have access to view data in your A/B testing framework. As a prerequisite of getting access, they should sit down for a half-hour training on statistical significance, minimum sample size, and who has authority to call tests.

If someone does peek, gently remind them about conventional testing methodology, and provide a specific time when you plan to check the test. By this point, you should know that, because you’ve calculated sample size before the test kicks off.

Shifting strategy too frequently

With rapid growth comes shifting strategy. If you’re caught unawares by strategy that changes too often, it means you’re not seeing enough into the inner workings of the business.

This is a common issue with anybody who’s hired as an outside contractor – but it’s an issue that any qualified consultant needs to fix, fast.

Shifting strategy is usually best answered by getting on a weekly call with executives. During the call, outline your progress, any blockers, future strategy, and a specific request for them to explain what they’re doing to grow the business right now.

Cancelling or abandoning meetings

Another issue with rapid-growth companies: people are too thinly stretched to give optimization the time and attention that it deserves. Usually, key stakeholders haven’t yet ceded control of operations that used to be within their purview.

Meetings are important because they help people gain consensus on strategy and tactics. The meetings we run are short and focused, rarely exceeding a half-hour – but they remain vitally important towards ensuring that everyone has their right marching orders.

Every time someone cancels a meeting on me – and it’s not because of a very obvious, potentially-business-destroying fire – I go over their head and express a concern. Mercifully, this does not happen often.

If it gets too bad, walk

You should always have a plan B in your career.

If you end up in a research-hostile environment that doesn’t value the right way to do things, do what you can to shift their mindset. If nothing changes, find a business that’s more amenable to your practice.

This is the final suggestion because it’s a last resort. Do what you can first. But don’t be afraid to walk.

← Back to the Blog Check Out Draft’s Store →