Limited Time Offer: Save 30% on PRO with code WOOHOO »

Real-World Education for Modern Marketers

Join Over 600,000 Marketing Professionals

Start here!
Text:  A A

What to Measure in Multivariate Testing: Defining Your Success Goals (Part 2)

by Eric Hansen  |  
June 10, 2008

In my last article, I defined multivariate testing and how it can optimize your Web marketing, as well as five common errors to avoid. Let's now look at what to measure in your tests and how to define your criteria for success.

Before you start formulating a test hypothesis, or begin running your tests, the first and most important step is to ensure that you have clearly defined objectives for your Web site. You'll want to examine your marketing goals in order to determine the appropriate success factors that all of your organization's stakeholders can agree upon.

Let's start with typical measureable Web site goals:

  1. Make money: sell product, generate leads and advertising or promotional click-throughs.
  2. Save money: enable users to adopt self-service features and answer product and service questions on their own (such as through online FAQs and documentation). 
  3. Create brand awareness and industry visibility.

It's important for all stakeholders to agree on the goals of your Web site, because when a decision is made to adjust or optimize something on the site, everyone's needs should be addressed. You'll want to make sure you are testing the things that truly matter for your organization and balancing performance across all stakeholders.

Once you determine the goals, the next step is sifting through potential key performance indicators (KPIs) to decide which will accurately measure progress toward your specific marketing goals and benchmarks. For example:

  • If your goal is to make money, you'll want to track those pages and areas of the site that users click on in the conversion funnel, such as the Buy Now button and resulting Thank You pages.
  • If your goal is to save money, track the interactions with both the self-service areas (like FAQs and help content) as well as non-self-service areas (like contact and help ticket generation) of your site.
  • Brand awareness goals can be more difficult to track, but certain KPIs can be proxies for customer loyalty, such as recency and frequency of visits and time spent on site, and the percentage of return visitors. Other behaviors to track could include "send to a friend" or "print page" features that indicate a visitor's interest in sharing your site with others.

Beware of scenarios where an increase in one desirable KPI can cause a decrease in another (perhaps more valuable) KPI. This cannibalization can sometimes be a Catch-22, so the best practice is simply to track both KPIs to provide increased visibility of user behavior.

Once you've agreed upon your Web site goals and KPIs, determine the goals of your multivariate testing strategy. These are often related to your Web site goals but can be much more granular.

Let's look at a typical retail/e-commerce Web site as an example. The owners of this site will obviously want their site to make money; here are some of the things they could test and measure in support of that goal:

  • Account or newsletter registration: Which navigation path successfully led to the most completions of a new account registration, newsletter signup, or other lead-generation form?
  • E-commerce: What elements of the Web site led to the most "Add to Cart" clicks, followed by successful order completion pages (e.g., "Thank you for your order")? Which combination of product information such as graphics, descriptions, layout, and color increased average order value?
  • Promotions/offers: Did the visitor click on a promotion, and then take the additional step(s) necessary to complete the conversion process? Bear in mind that it is not enough to simply track clicks from the promotion; if you do so, you risk optimizing for what is most persuasive in getting users to "click" one step further, but not necessary all the way to the conversion. What you need to do is measure both clicks and conversions, and in doing so you will determine the best way to get people to convert. You may end up with fewer people clicking on the promotion, but a higher percentage of them will convert, leading to a higher absolute number of conversions.

Sign up for free to read the full article.Read the Full Article

Membership is required to access the full version of this how-to marketing article ... don't worry though, it's FREE!


We will never sell or rent your email address to anyone. We value your privacy. (We hate spam as much as you do.) See our privacy policy.

Sign in with one of your preferred accounts below:


Eric Hansen is the president and founder of SiteSpect Inc. ( and architect of SiteSpect's multivariate testing and behavioral targeting platform. Reach Eric via

Rate this  

Overall rating

  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
6 rating(s)

Add a Comment


  • by s Thu Jun 12, 2008 via web

    Hi Eric
    you mention within the promotions/offers portion not only measuring clicks but conversions. How does one define conversions on a website where clicking on a certain promotion just gives you details on the existing promotion there is no real call to action on the website itself i.e. it is not geared towards providing e-commerce functionality. How would I define conversions in this sense?

    One more question :)

    The other question I had was on the “visit score” – how are the weights derived for the various variables i.e. c, a , s, & p each have weights 1,1,5 & 15 I was not sure how they were determined and how would I calculate a weighted value on my own accord?

MarketingProfs uses single
sign-on with Facebook, Twitter, Google and others to make subscribing and signing in easier for you. That's it, and nothing more! Rest assured that MarketingProfs: Your data is secure with MarketingProfs SocialSafe!