Out now: Value-Based Design, the definitive way to prove your design’s worth. Read it.

Heuristic Evaluation

 

A heuristic evaluation is a usability analysis of a site that you perform yourself – not by running a site past any paying customers. Essentially, you have a checklist (comprising the heuristics) that you run through – and then you take the findings from that checklist and translate them into actionable, revenue-generating design decisions.

Here are some typical things you do in a heuristic evaluation:

  • Perform a checkout with a real credit card. (Refund yourself later, of course.)
  • Search the site for products that fit a specific criteria.
  • Add upsell products to your cart.
  • Tweak sliders for a custom plan selector.
  • Run through these same tasks in a variety of browsers, on a variety of different device types. (Checking out on a smartphone with wifi & LTE turned off, for example.)

These are all obviously important things, but most team members never bother doing them once they’re set up and working. And your checkout form can aggressively leak revenue with random, pernicious bugs that exist only in contexts that you never bothered testing. Forcing your developers – or yourself – to take the time to run through common tasks can surface all sorts of conversion-killing bugs.

Heuristic evaluations are one of those things that you can do with a budget of $0 and no executive support. You can take the results of these, share them internally, and make a strong economic case for higher-touch optimization efforts.

Heuristic evaluations surface all sorts of major revenue-leaking issues with your website, like:

  • Your checkout form just flatly not working, for large segments of the population.
  • Mobile devices not loading the site well (or at all) on 2G or 3G connections.
  • Major issues with hierarachy & information architecture.
  • Byzantine conversion funnels full of tire-kickers who beeline for features and don’t bother using the product.

While it’s awesome that you can do this sort of thing yourself, there remains no substitute for similar sorts of tests with real-world, paying customers. Usability testing & interviewing happen in conjunction with heuristic evaluations, not instead of them. But heuristic evaluations are great for making a case for optimization internally – and then using that as leverage for more ambitious research efforts, even as you continue evaluating your site from time to time.

Here’s how you run a heuristic evaluation:

  1. Develop a checklist. Struggling to build one yourself? Great, the whole checklist I use with Revise Express clients is below. It’s intended for SaaS businesses, but almost all of it is repurposable for ecommerce as well.
  2. Run through each item on the checklist, and outline what’s working and what could stand to be improved. Ideally, you’ll be tying each one of these to specific revenue figures: for example, if you can show, in Google Analytics, that mobile is converting 30% worse than desktop, then that’s a strong business case for closely examining the usability of the mobile site, in a variety of devices and contexts.
  3. Write up a report that prioritizes various fixes. ConversionXL has three different frameworks that you can use for your own purposes. I use Trello boards for shuffling around various issues.

That’s it. How you act on a heuristic evaluation is up to you and the rest of your team. And while I heartily encourage you to take a different approach than my own, here is the exact heuristic evaluation that I run for all of my Revise Express clients:

Revise Express: Heuristic Evaluation Outline

Business & Positioning

  • Is a positioning statement articulated upfront?
  • Is it capable of attracting attention immediately?
  • Does it solve a concrete pain?
  • Does it have a specific customer in mind?
  • Does it focus on their needs?
  • How much of the text is focused on I/me/we, versus you/your/yours?

The Funnel

  • Is it clearly defined?
  • What pages does it comprise?
  • Can we eliminate any pages?
  • Are there any alternate entry points (e.g. landing pages)?
  • Are people making side tours (e.g. to features pages)? Are these people more or less likely to convert?

The Home Page

  • Is there social proof?
  • Is there a chatroll? Does the chatroll have a clearly defined purpose?
  • Is there a rotator/carousel? Does it fit each of the Baymard Institute guidelines for carousels (http://baymard.com/blog/homepage-carousel)?
  • Is there press proof beyond just logos indicating that coverage happened? Is the press proof for well-respected outlets (NYT, WSJ, FT, etc), or is it for outlets that act as press release factories (TechCrunch, TNW, etc)?
  • What does the masthead look like? Full-bleed? Stock photography or a real person? Does it lead with a headline?
  • Is the headline specific, attention-grabbing, and focused on alleviating the customer’s specific expensive problem?
  • Is there a subheader? What does it focus on next?
  • What links are in the header navigation? How does engagement vary among them? Can any of them be safely moved into the footer?
  • Is the CTA in the masthead?
  • Is the CTA the final element before the footer?
  • Is the footer clean & organized?
  • Do the home page’s modules focus on the benefits that the customer will get from the product, or do they focus on the product’s features?
  • Are the customer’s objections audited & addressed?
  • Is there a video? Where is it placed? What behavior does the video have on page load?

Calls to Action

  • How many of them are there?
  • Does each call to action focus explicitly on revenue generation?
  • Is there a clear delineation between primary & secondary calls to action?
  • How consistently treated are the calls to action?
  • Are the calls to action reworked from a theme default?
  • For modal calls to action, is there a “no” option?
  • How many modals are there? What behavior do they take (exit intent, time-delimited, etc)?
  • What call to action does each modal support?
  • Is a freebie given for email-based calls to action?
  • Are the calls to action intrusive or “sleazy”-looking?

Pricing

  • How clear is the pricing?
  • How easy is it to get to the pricing page?
  • How likely is it that people beeline for pricing?
  • Is there any upsell in the pricing section? Any “a la carte” options?
  • Is there any high-touch, “concierge”-based pricing?
  • Is the pricing organized into plan columns?
  • If so, do the columns have plan names, a most popular plan, a focus on differentiation between each plan, etc?
  • What does the objection-busting FAQ look like?
  • Is there a split between annual & monthly plans? Which one is the default? What happens when you swap between them?

Sign Up

  • Is this page different in any way from the login screen?
  • Is this page different in any way from the default-styled signup form for your given framework?
  • Are objections still addressed here? Is there any “follow-through”?
  • What happens if errors occur?

The Checkout Process

  • Does credit card number, zip code (optional, for USA only), CVV/CVC/CID, and expiration date show up with the numeric keypad on mobile?
  • Is autocorrect turned off for name, address, and city?
  • Is the country selector geolocated?
  • Does the email keyboard appear for the email field on mobile?
  • How is the country & state selectors’ behaviors?
  • Does the logo of the credit card issuer (Visa/MC/Amex/Discover) appear when you begin to type it in?
  • Does CVV/CVC/CID show in a diagram on an example credit card after its corresponding type is selected? (Each one has a different acronym, and placement & validation is different for Amex.)
  • Is the usability of the credit card form in line with industry standards (cf. creditcardjs.com)?
  • Does the padlock appear in the browser bar, indicating HTTPS?
  • Are you punted to a third party checkout solution (2CO, Shopify, MoonClerk, PayPal)? Is this in conjunction with a more conventional checkout? What’s the customer breakdown between each checkout flow? Is conversion helped or harmed by the third-party checkout provider?
  • Are there multiple ways to check out for multiple different credit card issuers (e.g. Amex through PayPal, Visa/MC through Authorize.net)?
  • Do you accept payment by check or wire transfer? Under what conditions? How do you handle recurring payments?

Per-Browser Issues

Vet GA for all browsers. Are any browsers not converting well? Assess why by loading the site in that particular browser, determining page load time relative to the mean (which you can get in GA), and aggressively bugfixing for it.

Per-Device Issues

Vet GA for all devices. Are any devices not converting well? Assess why by getting ahold of an example device, and loading the site in it.

Mobile Issues

  • Does the home page render well on mobile?
  • Is there a significant conversion difference between smartphone, tablet, and desktop? Figure out why and what you can do to fix it.
  • Does the page show a “hamburger menu”? If so, what does that look like when opened?
  • How does pricing look on mobile?
  • How does the page load on 3G?
  • Is page load time severely impacted for mobile devices, and if so, what can we do to make it render at parity with desktop/wireline connections?
  • Are people ready to make a purchasing decision on mobile? If not, can we create a funnel that directs them towards some other action?
  • Is the checkout form responsive?
  • What happens if the internet connection is lost mid-signup? What happens to onboarding?

How do I run through these?

I evaluate exactly enough to comprehensively answer the question. That might involve a lot of further investigation, more questions, and other points to evaluate. Let’s take a few examples:

Does the home page render well on mobile?

You have to try this on a variety of mobile devices. I use iPhones 5, 6, and 6 Plus. You can simulate these in Google Chrome natively if you want.

I also try to use a variety of connection speeds. I turn off wifi and LTE by default – and then play around on 3G. I even go into the basement of my house (where the reception is awful – early 1900s, 1’ thick foundation, etc) and try there.

Is the site responsive? Does the page load quickly? Does document.ready happen early enough that you can begin taking action, or is there some sort of asset that’s holding the rest of it up?

If you’re a B2B app, what’s the next step? People might not be as likely to sign up when they’re viewing your site on the train – so maybe capturing their email address for a demo or sales call is the most salient call to action. Is it?

Have you even tried this with your own business’s website lately? If so, count yourself among the lucky few who actually care about how real-world people are using your site.

Third-party checkout solutions

In the heuristic:

Are you punted to a third party checkout solution (2CO, Shopify, MoonClerk, PayPal)? Is this in conjunction with a more conventional checkout? What’s the customer breakdown between each checkout flow? Is conversion helped or harmed by the third-party checkout provider?

I would dive into Google Analytics and outline the breakdown in conversion rates among each checkout provider. I would also ask about the motivations for adding a third-party provider. Was it because customers asked for it? Did customers actually take action on it?

Some people depend on PayPal for the livelihood of their business. Others get their absolute worst customers through PayPal. You need to determine which kind of business you are. Perhaps removing it really does make sense.

Are the customer’s objections audited & addressed?

What are the customer’s objections? Do we have a clear sense of them? If not, run a survey and ask what held them up from closing the sale.

Where is it appropriate to address their objections? On an ecommerce site, it’s usually on the product page. For a SaaS, it’s usually on the home page.

And so on

I’ve spent a lot of time harping on the bug-fixing side of optimization here. But you’re also coming up with ideas for revenue-generating design decisions to run in A/B tests.

For example, I’ve run heuristic evaluations in the form of Revise Express reports for dozens of businesses. With SaaS pricing grids, I’ve found that people hew too closely to the four-column layout – which amounts to the default for SaaS these days.

Reworking this area often provides 10% revenue bumps for these businesses. How? By noticing that most people are favoring specific plans, or are more likely to have a long-term sales process at higher-value plans – and reworking the pricing to reflect real-world customer behavior.

And that’s just one example. With heuristic evaluations, you can step back and audit your site with a clear head. Run a heuristic evaluation once every three months to ensure that your site remains as optimally focused on conversion as possible.

← Back to the Blog Check Out Draft’s Store →