New: Draft Membership. Level up your store’s design game – and profit. Read more.

Heuristic Evaluation

 

A heuristic evaluation is a usability analysis of a site that you perform yourself – not by running a site past any paying customers. Essentially, you have a checklist (comprising the heuristics) that you run through – and then you take the findings from that checklist and translate them into actionable, revenue-generating design decisions.

For this lesson, we'll talk about how to do a heuristic evaluation, then talk about times when it falls short- and what to do then.

Here are some typical things you do in a heuristic evaluation:

  • Perform a checkout with a real credit card. (Refund yourself later, of course.)
  • Search the site for products that fit a specific criteria.
  • Add upsell products to your cart.
  • Tweak sliders for a custom plan selector.
  • Run through these same tasks in a variety of browsers, on a variety of different device types. (Checking out on a smartphone with wifi & LTE turned off, for example.)

These are all obviously important things, but most team members never bother doing them once they’re set up and working. And your checkout form can aggressively leak revenue with random, pernicious bugs that exist only in contexts that you never bothered testing. Forcing your developers – or yourself – to take the time to run through common tasks can surface all sorts of conversion-killing bugs.

Heuristic evaluations are one of those things that you can do with a budget of $0 and no executive support. You can take the results of these, share them internally, and make a strong economic case for higher-touch optimization efforts.

Heuristic evaluations surface all sorts of major revenue-leaking issues with your website, like:

  • Your checkout form just flatly not working, for large segments of the population.
  • Mobile devices not loading the site well (or at all) on 2G or 3G connections.
  • Major issues with hierarachy & information architecture.
  • Byzantine conversion funnels full of tire-kickers who beeline for features and don’t bother using the product.

While it’s awesome that you can do this sort of thing yourself, there remains no substitute for similar sorts of tests with real-world, paying customers. Usability testing & interviewing happen in conjunction with heuristic evaluations, not instead of them. But heuristic evaluations are great for making a case for optimization internally – and then using that as leverage for more ambitious research efforts, even as you continue evaluating your site from time to time.

Here’s how you run a heuristic evaluation:

  1. Develop a checklist. Struggling to build one yourself? Great, the whole checklist I use with Revise Express clients is below. It’s intended for SaaS businesses, but almost all of it is repurposable for ecommerce as well.
  2. Run through each item on the checklist, and outline what’s working and what could stand to be improved. Ideally, you’ll be tying each one of these to specific revenue figures: for example, if you can show, in Google Analytics, that mobile is converting 30% worse than desktop, then that’s a strong business case for closely examining the usability of the mobile site, in a variety of devices and contexts.
  3. Write up a report that prioritizes various fixes. ConversionXL has three different frameworks that you can use for your own purposes. I use Trello boards for shuffling around various issues.

Develop Your Own Checklist

First, break down each type of page in your funnel

For most stores, this means you have:

  • Home
  • Collection
  • Product Detail
  • Cart
  • Checkout
  • Search results
  • Thank you/order status

Then, break out areas in each page

Areas are general sections that correspond to a given focus on your store. In Shopify, this roughly corresponds to individual sections, especially on your home & product detail pages. Some of these, especially navigation, are repeated across your store. Each of these areas contains one or many elements, which we’ll talk about in a minute.

For example, on your home page, you probably have:

  • Desktop nav
  • Mobile nav
  • Masthead
  • Description
  • Cross-selling
  • Mailing list callouts
  • Popovers

And on your product detail page, you probably have:

  • Desktop nav
  • Mobile nav
  • Image carousel
  • Buy box
  • Description
  • Cross-selling
  • Reviews

You get the idea.

Next, identify each element in each area

Then, you’ll go into each area and audit the elements in all of them. For a home page masthead, that includes:

  • The background image
  • The headline
  • The call to action button
  • Call to action text

That’s a simple example. Your product detail page’s buy box includes:

  • Product name
  • Review average & count
  • Price
  • ATC button
  • Quantity selector
  • Variant selector(s)
  • Availability
  • A short description, maybe

Revise Express: Heuristic Evaluation Outline

While I heartily encourage you to take a different approach than my own, here is the exact heuristic evaluation that I run for all of my Revise Express clients:

Business & Positioning

  • Is a positioning statement articulated upfront?
  • Is it capable of attracting attention immediately?
  • Does it solve a concrete pain?
  • Does it have a specific customer in mind?
  • Does it focus on their needs?
  • How much of the text is focused on I/me/we, versus you/your/yours?

The Funnel

  • Is it clearly defined?
  • What pages does it comprise?
  • Can we eliminate any pages?
  • Are there any alternate entry points (e.g. landing pages)?
  • Are people making side tours (e.g. to features pages)? Are these people more or less likely to convert?

The Home Page

  • Is there social proof?
  • Is there a chatroll? Does the chatroll have a clearly defined purpose?
  • Is there a rotator/carousel? Does it fit each of the Baymard Institute guidelines for carousels (http://baymard.com/blog/homepage-carousel)?
  • Is there press proof beyond just logos indicating that coverage happened? Is the press proof for well-respected outlets (NYT, WSJ, FT, etc), or is it for outlets that act as press release factories (TechCrunch, TNW, etc)?
  • What does the masthead look like? Full-bleed? Stock photography or a real person? Does it lead with a headline?
  • Is the headline specific, attention-grabbing, and focused on alleviating the customer’s specific expensive problem?
  • Is there a subheader? What does it focus on next?
  • What links are in the header navigation? How does engagement vary among them? Can any of them be safely moved into the footer?
  • Is the CTA in the masthead?
  • Is the CTA the final element before the footer?
  • Is the footer clean & organized?
  • Do the home page’s modules focus on the benefits that the customer will get from the product, or do they focus on the product’s features?
  • Are the customer’s objections audited & addressed?
  • Is there a video? Where is it placed? What behavior does the video have on page load?

Calls to Action

  • How many of them are there?
  • Does each call to action focus explicitly on revenue generation?
  • Is there a clear delineation between primary & secondary calls to action?
  • How consistently treated are the calls to action?
  • Are the calls to action reworked from a theme default?
  • For modal calls to action, is there a “no” option?
  • How many modals are there? What behavior do they take (exit intent, time-delimited, etc)?
  • What call to action does each modal support?
  • Is a freebie given for email-based calls to action?
  • Are the calls to action intrusive or “sleazy”-looking?

Pricing

  • How clear is the pricing?
  • How easy is it to get to the pricing page?
  • How likely is it that people beeline for pricing?
  • Is there any upsell in the pricing section? Any “a la carte” options?
  • Is there any high-touch, “concierge”-based pricing?
  • Is the pricing organized into plan columns?
  • If so, do the columns have plan names, a most popular plan, a focus on differentiation between each plan, etc?
  • What does the objection-busting FAQ look like?
  • Is there a split between annual & monthly plans? Which one is the default? What happens when you swap between them?

Sign Up

  • Is this page different in any way from the login screen?
  • Is this page different in any way from the default-styled signup form for your given framework?
  • Are objections still addressed here? Is there any “follow-through”?
  • What happens if errors occur?

The Checkout Process

  • Does credit card number, zip code (optional, for USA only), CVV/CVC/CID, and expiration date show up with the numeric keypad on mobile?
  • Is autocorrect turned off for name, address, and city?
  • Is the country selector geolocated?
  • Does the email keyboard appear for the email field on mobile?
  • How is the country & state selectors’ behaviors?
  • Does the logo of the credit card issuer (Visa/MC/Amex/Discover) appear when you begin to type it in?
  • Does CVV/CVC/CID show in a diagram on an example credit card after its corresponding type is selected? (Each one has a different acronym, and placement & validation is different for Amex.)
  • Is the usability of the credit card form in line with industry standards (cf. creditcardjs.com)?
  • Does the padlock appear in the browser bar, indicating HTTPS?
  • Are you punted to a third party checkout solution (2CO, Shopify, MoonClerk, PayPal)? Is this in conjunction with a more conventional checkout? What’s the customer breakdown between each checkout flow? Is conversion helped or harmed by the third-party checkout provider?
  • Are there multiple ways to check out for multiple different credit card issuers (e.g. Amex through PayPal, Visa/MC through Authorize.net)?
  • Do you accept payment by check or wire transfer? Under what conditions? How do you handle recurring payments?

Per-Browser Issues

Vet GA for all browsers. Are any browsers not converting well? Assess why by loading the site in that particular browser, determining page load time relative to the mean (which you can get in GA), and aggressively bugfixing for it.

Per-Device Issues

Vet GA for all devices. Are any devices not converting well? Assess why by getting ahold of an example device, and loading the site in it.

Mobile Issues

  • Does the home page render well on mobile?
  • Is there a significant conversion difference between smartphone, tablet, and desktop? Figure out why and what you can do to fix it.
  • Does the page show a “hamburger menu”? If so, what does that look like when opened?
  • How does pricing look on mobile?
  • How does the page load on 3G?
  • Is page load time severely impacted for mobile devices, and if so, what can we do to make it render at parity with desktop/wireline connections?
  • Are people ready to make a purchasing decision on mobile? If not, can we create a funnel that directs them towards some other action?
  • Is the checkout form responsive?
  • What happens if the internet connection is lost mid-signup? What happens to onboarding?

How do I run through these?

For every single element that you’ve identified, you need to exhaustively describe what makes that element perform as well as it possibly can. I evaluate exactly enough to comprehensively answer the question. That might involve a lot of further investigation, more questions, and other points to evaluate. Let’s take a few examples:

Does the home page render well on mobile?

You have to try this on a variety of mobile devices. I use iPhones 5, 6, and 6 Plus. You can simulate these in Google Chrome natively if you want.

I also try to use a variety of connection speeds. I turn off wifi and LTE by default – and then play around on 3G. I even go into the basement of my house (where the reception is awful – early 1900s, 1’ thick foundation, etc) and try there.

Is the site responsive? Does the page load quickly? Does document.ready happen early enough that you can begin taking action, or is there some sort of asset that’s holding the rest of it up?

If you’re a B2B app, what’s the next step? People might not be as likely to sign up when they’re viewing your site on the train – so maybe capturing their email address for a demo or sales call is the most salient call to action. Is it?

Have you even tried this with your own business’s website lately? If so, count yourself among the lucky few who actually care about how real-world people are using your site.

Third-party checkout solutions

In the heuristic:

Are you punted to a third party checkout solution (2CO, Shopify, MoonClerk, PayPal)? Is this in conjunction with a more conventional checkout? What’s the customer breakdown between each checkout flow? Is conversion helped or harmed by the third-party checkout provider?

I would dive into Google Analytics and outline the breakdown in conversion rates among each checkout provider. I would also ask about the motivations for adding a third-party provider. Was it because customers asked for it? Did customers actually take action on it?

Some people depend on PayPal for the livelihood of their business. Others get their absolute worst customers through PayPal. You need to determine which kind of business you are. Perhaps removing it really does make sense.

Are the customer’s objections audited & addressed?

What are the customer’s objections? Do we have a clear sense of them? If not, run a survey and ask what held them up from closing the sale.

Where is it appropriate to address their objections? On an ecommerce site, it’s usually on the product page. For a SaaS, it’s usually on the home page.

And so on

I’ve spent a lot of time harping on the bug-fixing side of optimization here. But you’re also coming up with ideas for revenue-generating design decisions to run in A/B tests.

For example, I’ve run heuristic evaluations in the form of Revise Express reports for dozens of businesses. With SaaS pricing grids, I’ve found that people hew too closely to the four-column layout – which amounts to the default for SaaS these days.

Reworking this area often provides 10% revenue bumps for these businesses. How? By noticing that most people are favoring specific plans, or are more likely to have a long-term sales process at higher-value plans – and reworking the pricing to reflect real-world customer behavior.

And that’s just one example. With heuristic evaluations, you can step back and audit your site with a clear head. Run a heuristic evaluation once every three months to ensure that your site remains as optimally focused on conversion as possible.


However there are times when heuristic evaluations fall short. Let's talk about those.

Where does heuristic evaluation fall short, and what research methods should you employ, instead?

Heuristic evaluation is great, and everybody should do it. It fixes the biggest issues. It outlines one-off fixes better than almost any other research method. It’s maybe the best research method from a cost-benefit standpoint, making it also the lowest risk research method. At Draft, we run heuristic evaluations all the time. I ran one right before writing this lesson.

Yet as with any research method, there are limits to heuristic evaluations – or we’d just be a heuristic evaluation consultancy. Feels boring, right? Let’s talk about all of the questions that heuristic evaluation can’t answer – and what research methods you should use, instead.

What your customers are actually doing on your store

A heuristic evaluation is subjective to your own experience, and your customers aren’t creating your heuristic checklists. As a result, you need different research methods to understand how customers are behaving on your live store.

Instead, use: heat & scroll maps, analytics.

How your customers feel

Again, you have no idea how your customers respond to your messaging, copy, branding, or overall ethos as a company. The only way to accomplish this is by asking them.

Instead, use: interviews.

Whether your customers have different abilities from you

You know too much about your store. You know too much about how technology works. You know too much about your products. Your customers know as much as they care to know. If you want to understand where your store’s usability truly falls down, then you need to watch your customers go through tasks themselves, not yourself. This will uncover many issues that your evaluations are failing to.

Instead, use: usability tests, post-purchase surveys.

What problems your customers want solved

The purchasing process is holistic, continuing past conversion and on to delivery & use. You care about making sure customers buy. Your customers care about whether your product looks like the photos, whether it fits well, whether it does what it says it’ll do. What objections can you address before customers convert? When do those objections appear? No heuristic evaluation is capable of answering this.

Instead, use: customer support inquiry analysis, post-purchase surveys.

← Back to the Blog Check Out Draft’s Store →