Real-World Education for Modern Marketers

Join Over 600,000 Marketing Professionals

Start here!
N E X T
Text:  A A

What to Test in Multivariate Testing: Identifying Site Factors (Part 3)

by   |    |  5,046 views

In my previous article, I looked at defining your success goals and what to measure when running multivariate tests. Let's now look at your site factors and learn how to select the right ones to test.

By now, your marketing goals are clearly defined and you're ready to run a multivariate test to optimize your site's marketing effectiveness... but which elements, or factors, should you test?

What Is a Site Factor?

First, let's define what a factor is. A site factor is a distinct, single element that you can control for testing purposes, and each version of a factor is called a variation. Through testing, you'll discover that some factors are influential and have a causal relationship with user behavior—modifying them changes how users behave, ideally motivating them to reach your marketing goalposts (e.g., checkout, registration, or simply spending more time on your site.) Others factors are not influential, and have little to no impact on what users do.

One of the keys to multivariate testing is in selecting factors to test, and defining them so that they may be accurately analyzed.


Many unique factors exist on every site, and they vary in scope from narrow to broad. For example, a factor could be an element on a single page, such as a headline, image, or paragraph of copy. Or, it might span pages or exist across a particular section, such as a navigation component, product category promotion, or a call-to-action. As well, a site factor could span your entire site, such as a text style (CSS), or grid layout spacing (e.g., how much space is allocated for main content vs. advertising vs. widget areas, etc.).

Choosing Site Factors

If multivariate testing gives us the ability to simply throw everything at the wall to see what sticks, what's the big deal with choosing specific factors to focus on? Why not just test everything and anything? While identifying factors isn't that hard, coming up with a broad range of variations takes time and effort. For example, think of the creative resources required for coming up with five variations for each of 20 different site factors, which could include everything from the page title to the copyright in the footer. While it's technically possible to test it all factors and their variations at once, it's much easier to be organized and test fewer factors that have a higher likelihood of actually making a difference.

By thinking like an end user and navigating through your site, you can uncover the main factors that are directly in the paths through which your users make their way toward reaching your marketing goals. What are the words that you have to read, the images that you must view, and the links that you're supposed to click on?

These touch points are your low-hanging fruit and are often the factors that, when optimized, will give you the biggest ROI because they are encountered in every meaningful task or process. A few more specific examples include registration forms, product presentation and layout, checkout funnels, and high visibility slots used for offers, cross-sells, or ad space.

Have a Clear Hypothesis

Referring back to my previous article, whenever you are running a test, make sure you have a clear, defined hypothesis. This is nothing more complicated than filling in the blanks in the following statement: "What would happen to ___ if we did ___?" For example, what would happen to newsletter opt-in rates if we changed the privacy headline? This simple yet important step ensures that you know what you are testing, and how you'll determine whether any changes increased performance.

Properly Isolating Your Site Factors

When defining factors on your site for testing, be careful of accidently creating factors that actually consist of two or more factors. For example, most headlines are at least four factors: words, font, color, and size. This means that when defining a factor and producing variations of it for a test, you should change only one thing per variation.

To elaborate, let's work through an example of what not to do, and see what happens if we were to test these two alternate headlines: "buy one get one free" in red, and "two for the price of one" in blue.

Here we have two variations, but we're incorrectly mixing the word change with the color change. This obscures our ability to determine whether either of those factors caused users to behave differently. If we were to analyze the results of that test and learn that "two for the price of one" in blue outperformed its rival (increased conversion rate), then how do we know if it was the words or the color that made the difference? Or, what if the two variations performed the same—couldn't it have been that the words performed better, while the change in color performed equally worse. In other words, one change helped but the other hurt, and the two canceled each other out. Properly isolating the two factors—words and color—avoids these problems and lets you cleanly analyze the effects of each factor.

Flexing Your Creative Muscle

Once you have identified and isolated promising factors on your site, it's time to get the creative juices flowing. This is perhaps the essence of the testing process—coming up with variations of your site that might be better at persuading, easier to use, and more engaging to your users.

The best-practice here is to make sure that there are considerable differences across the variations that you create for each factor. Doing so ensures that you are properly "exercising" each factor to determine whether changing it has any material impact on how users behave. Subtle differences between variations are likely to result in subtle changes in behavior, so don't be afraid to try polar opposite variations—this is a test, after all.

Examples of Common Site Factors

Now let's review a punch list of common site factors. Each site is unique and must be independently tested and optimized, but the following is a list of common factors on most every site. Use this list as a starting point for zeroing in on your site's influential factors:

Text/copy

  • Long versus short
  • Style and tone, such as chatty versus formal
  • Positioning—which value proposition, features, and benefits work best?
  • Call-to-action text

Font, color, and size

  • Think about your target audience (e.g., 10% of men are colorblind)
  • If your audience is older, test larger fonts

Buttons

  • Color
  • Location
  • Copy

Navigation

  • Sequence of items
  • Labels (e.g., "My profile" vs. "Profile" vs. "Settings")
  • Inline text links
  • Link style: underline, bold, color, etc.

Images

  • Content of image (e.g., man vs. woman vs. group)
  • Size
  • Location on page

Layout

  • Location and size of areas/boxes on the page
  • Attention focus—what layouts help focus vs. create distraction?
  • Where should you place buttons and links?

Functionality

  • 3-page checkout vs. 5-page checkout
  • Forms: required fields vs. optional fields
  • That new site bell/whistle... is it helping or hindering?

* * *

Next time, I'll talk about targeting user segments and how to test and optimize distinct audiences. If you have any questions or comments in the meantime, feel free to reach out to me at ehansen@sitespect.com.  


Join over 600,000 marketing professionals, and gain access to thousands of marketing resources! Don't worry ... it's FREE!

WANT TO READ MORE?
SIGN UP TODAY ...
IT'S FREE!

We will never sell or rent your email address to anyone. We value your privacy. (We hate spam as much as you do.) See our privacy policy.

Sign in with one of your preferred accounts below:

Loading...

Eric Hansen is the president and founder of SiteSpect Inc. (www.sitespect.com) and architect of SiteSpect's multivariate testing and behavioral targeting platform. Reach Eric via ehansen@sitespect.com.

Rate this  

Overall rating

  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
3 rating(s)

Add a Comment

MarketingProfs uses single
sign-on with Facebook, Twitter, Google and others to make subscribing and signing in easier for you. That's it, and nothing more! Rest assured that MarketingProfs: Your data is secure with MarketingProfs SocialSafe!