How do you know what to do next? Every optimization program has a common thread: making sense of all of the design decisions you could test, and sorting them in an order that makes sense for the business.

You need a way to sort new design decisions consistently. In doing so, individual egos take a back seat, flattening the org chart and embracing rational decision-making.

Obviously, we have our own process. Here’s what you can do to prioritize new test ideas quickly.

Experiments

In order to make sure you’re prioritizing sensibly, you should dive into each design decision and understand how easy it is to make, what potential impact it might have, and how well it’s aligned with existing research.

Each of these parameters is scored from 0 to 10. Then you add them up, and sort descending by score.

Parameter 1: Feasibility

If you’re building it yourself

The easiest tests are the ones that you can build yourself in your experimentation framework or repository.

Start from 10, representing the easiest possible implementation.

• If you’re making strictly copy changes or removing semantically scoped elements, score a 10. You can do this yourself in your sleep.
• If you’re rearranging elements, incorporating different sets of changes for mobile & desktop, or writing any CSS to hide or unhide specific features or dynamic functionality, score a 9.
• If you have to incorporate JavaScript in your prototype, score a 8.

If you’re working with others

If you need to enlist others to build out any server-side functionality, start at 8, and subtract a point for each of the following that’s true, stopping at 0:

• Separate versions need to be built for mobile & desktop.
• Dynamic functionality needs to be built for every product in a collection page.
• Filtering or sorting elements are affected in any way other than removal.
• New standard-issue elements are being added to a page.
• New scarcity plays are being added to a page.
• Simple dynamic functionality, like a countdown timer to a static time, is being added to a page.

And subtract two points for any of the following that’s true:

• Specific features need to be rolled out for wildcard pages.
• A high-level framework needs to be built out, such as a feature flag.
• Elements are affected that have direct functional ramifications for the customer’s transaction.
• Multiple pages need to be adapted to handle edge cases or address a consistent customer experience.
• Significant portions of technical debt need to be navigated.

Parameter 2: Impact

1. Start with 5 points.
2. Go through each group of questions, and add & subtract points accordingly.
3. Discard the idea if you score below a 7.
4. Prioritize it like you normally would if you score a 7 or above.

Set 1: add a point for yes, subtract a point for no

• Will over 80% of customers see the change?
• Does the change apply to a segment of customers that converts at least 1/3 below the site average?
• Is the change mobile first?
• Is the change significant enough to change over half of the page’s elements?
• Is the element located above the fold on mobile?
• Does the element map directly to customer motivations?
• Does the test occur on a page that’s within the conversion funnel (home, collection, product, cart, checkout)?
• Does the change specifically map to a purchasing decision?

Set 2: add a point for yes, do nothing for no

• Does the element promote scarcity?
• Does the element promote increased customer lifetime value?
• Does the element simplify the page?
• Is the element noticeable within 1 second on average on behavior recordings?
• Has a previous experiment been run on this element?

Set 3: add a point for yes, subtract 2 points for no

If you answered “yes” to the previous question, did the experiment succeed or fail?

• Succeeded: add 2 points.
• Failed: add 1 point.
• Inconclusive/null result: subtract 2 points.

Parameter 3: Research Alignment

For each new decision, start at a score of 0 out of 10.

• Add 3 points if the decision is supported by more than 1 customer interview participant.
• Add 3 points if the decision is supported by at least one round of usability testing.
• Add 2 points if the decision is supported by post-purchase surveys.
• Add 2 points if the decision is supported by customer support.
• Add 1 point if the decision is supported by competitive analysis.
• Add 1 point if the decision is supported by behavior recordings.
• Add 1 point if the decision is supported by heat & scroll maps.

Cut any decisions that score a 4 or below.

One-off changes

All one-off changes are prioritized by three parameters:

• Feasibility
• Impact
• Context fit

Context fit

Context fit is defined by whether a new or changed element stylistically connects to the elements around it. Some elements should stand out – in a way that is consistent.

Start with 10 points.

• Does the changed element use the same styling as the rest of the layout? If no, deduct 2 points.
• Does the changed element use the same typography as the rest of the layout? If no, deduct 2 points.
• Does the changed or new element make a substantial change to the layout across devices? If yes, deduct 2 points.
• Does the changed or new element make a substantial change to the interaction model? If yes, deduct 2 points.
• If applicable, does the copy in the element fit the voice & tone of the rest of the business? If no, deduct 2 points.
• If significant changes are being made, what is the conversion- or usability-focused justification for them? If there is significant justification for each deduction, add 1 point.

Some design decisions might need to be changed to fit context. Err on the side of changing a design decision to improve this score.

← Back Store →