New: Draft Membership. Level up your store’s design game – and profit. Read more.

Usability Tests

Usability testing is what happens when you turn the interview process outward, by asking paying customers to complete specific tasks. Then, you time their responses, gauge their reaction, and ask them to explain their impressions.

Here are a few example tasks:

  • Add an item to your cart.
  • Sign up for a plan that best matches your needs.
  • Check out using a fake credit card (or a real one, and we’ll refund you).
  • Find all blue products.
  • Search for all products under $50 with a 4 star average review.
  • Search for all accessories for a given product.
  • Sign up for a given plan, and walk me through your onboarding.

Usability tests prove the effects of design decisions quantitatively by timing the user, counting a task’s steps, or assessing the number of errors they make. If the time (or number of steps) decreases to complete a task, we say that the product’s usability has increased by that much. Effective tests need a way to ensure consistency between test subjects, as well as a way to measure success.

The bible of usability testing is Don’t Make Me Think! by Steve Krug. It was first released in 2000 and still remains fresh, having been frequently updated to match new technological developments. It’s an essential, entertaining text that summarizes exactly how to improve your site’s usability.

Before we dive in, I will repeat two foundational points I’ve made in the past:

  • Research is incomplete without qualitatively assessing the performance of your site with real-world, paying customers. The absolute best way to do this is by talking directly to them in a 1-on–1 conversation. No matter how nerdy you might be (and believe me, I can identify), there is no substitute for basic human communication.
  • A/B testing is fundamentally rooted in customer-focused research methods.

Still here? Great. Let’s do this.

The fundamentals

First, you need to recruit the participants for the test. You need paying or prospective customers; coworkers come with too much baggage about how the product works. Usability tests usually require at least 3 participants, but not more than 5. Why at least 3? Because you want to make sure you aren’t accidentally recruiting an outlier. Why fewer than 5? You’ll get diminishing returns on your insights, and it’ll take too long to synthesize.

Once you have your participants, give them an example task – or tasks – to complete. You need to set up the right expectations for them, though, so you can get as many valuable insights as possible. The customer should vocalize their thought process as much as possible, out loud, so you understand when they’re being thrown off by something.

Record the participant’s screen, camera, and microphone for all tests. The best way to do this is with ScreenFlow. Remotely, have them screen share over Skype – and then run ScreenFlow on your end.


Once you have your research together, how do you translate it all into revenue-generating, testable design decisions?

  1. Clearly identify what isn’t working. Are people pausing at a given page for too long? Is there any confusion around a given element? Is form validation not working up to spec? Audit it and throw it in a Trello board or other issue-tracking tool, so it can be fixed. The Nielsen/Norman Group has a lot more on this topic.
  2. Outline qualitative findings. Are people providing open-ended questions? Are they vaguely confused or unenthusiastic about completing the task? These should be audited as well; it may be that you can address them in conjunction with survey results, 1-on–1 interviews, or even heat maps. has a whole article about how to record usability findings.
  3. Fix bugs. Did someone encounter a bug? Fix it. Prioritize it. It’s happening at the point where someone is about to give you money, making it vitally important for your business.
  4. Test copy. Is a call to action ambiguously worded? Any fixes should be tested, so you can determine their economic impact.
  5. Test layout & behavior. Are you changing the interaction model or layout of your revenue-generating pages? These changes should be tested – so you can understand whether the changes are capable of providing the lift necessary to confirm that you’ve fixed the usability issue at play.

Conversion Crimes: the easiest way to run a usability test

Conversion Crimes is the easiest way to run a usability test. They handle recruitment and expectation-setting. You get a video back for each participant. along with a summary of how each question was answered..

I strongly recommend Conversion Crimes for any applications that would skew “conventional” (e.g. not enterprise sales, not insanely niche positioning, etc) or any ecommerce sites with a broad B2C focus. Conversion Crimes doesn’t work so well for particularly wonky tasks, hellaciously focused recruitment strategies, aggressive market segmentation, or other such advanced-level techniques. It’s good for surfacing the biggest and most important conversion killers on your site, though, and that’s absolutely valuable for pretty much any business.

Are you a business with enough revenue-generating transactions to support A/B testing? Great. Sign up for Conversion Crimes and ask five people to sign up with a dummy credit card. If you ship fixes for the usability issues that uncovers, and you don’t make back Conversion Crimes’ fees within a month, I will literally eat a hat.

A case study

I once had a Draft Revise client that sold programming tutorial screencasts. They had four(!) calls to action, three of which were revenue-generating:

  • Signing up for an account (free)
  • Upgrading your account to a pro account ($)
  • Buying a team plan, for many pro accounts ($$)
  • Onsite enterprise training for your entire team, where instructors fly in and do an all-day workshop ($$$)

Team plans were pretty good reward for hardly any work. Enterprise training was high reward, but you’d have to arrange flights, take a week off your job, etc. There was a strong business case for selling people on team plans; after all, teams on this service would be more likely to splash out for enterprise training.

We ran a usability test on UserTesting, asking 5 participants to find and sign up for the team plan. None did. In fact, three of them beelined for the enterprise training and said “okay! I found the team plan!” This is obviously undesirable for us, and running the test exposed significant issues in the layout & wording of each call to action. We did 3 things:

  • Reworked the team plan’s call to action in the header navigation.
  • Provided a side-sell for the team plan on the enterprise training’s marketing page.
  • Created a “convince your boss” program for getting your whole team on the site.

Frequent mentions of the team plan could occur in lifecycle emails, the business’s blog, and so on – in order to keep it top-of-mind that team plans exist, and you always have the option to upgrade to them.

Usability testing is low-risk and high-reward

Especially when done through TryMyUI, usability testing requires very little resources in an organization. You can fit 5 usability tests on most businesses’ petty cash cards (which commonly have $500 limits in the US). You can use TryMyUI to limit the amount of time spent on recruitment & execution.

And I’ve never, in over 10 years of doing this, seen a usability test that didn’t generate significant revenue-generating insights for a business. Hopefully you’ve gained a sense of why conversion-focused usability testing is important, and how to execute it in your organization.

← Back to the Blog Check Out Draft’s Store →